Evaluation of the Early Impact of the UCLA/UCSF ACEs Aware Family Resilience Network (UCAAN)

In this study, we present results and recommendations based on an evaluation of the impact of the UCLA/UCSF ACEs Aware Family Resilience Network (UCAAN) on its goals of promoting screening for Adverse Childhood Experiences (ACEs), such as abuse and neglect, and trauma-informed health care (TIHC) for Medi-Cal beneficiaries. UCAAN was launched in 2021 as a multi-campus initiative leveraging the combined resources of the University of California, Los Angeles (UCLA); the University of California, San Francisco (UCSF); and the University of California, Davis. Since then, UCAAN has grown into a statewide network of organizations that work together to achieve its goals. UCAAN contracted with RAND to assess the impacts of its work and inform future policy decisions and resource investments.

Major Findings

UCAAN Has Dramatically Increased the Clinical Capacity to Screen for ACEs

A key way that UCAAN fosters clinical skills in ACE screening and TIHC is the Becoming ACEs Aware (BAA) course, a free online training that certifies Medi-Cal providers to be reimbursed for ACE screening. The BAA training has been taken by nearly 45,000 individuals, many of whom provide medical care to Medi-Cal beneficiaries. The training continues to be completed at a steady rate each month, suggesting continued growth in the knowledge and skills of the provider community. The Medi-Cal clinicians who have taken the BAA are diverse with respect to medical specialty, profession, and region of the state. In evaluations completed immediately after the trainings, ratings of the BAA's quality were very high.

We conducted the first follow-up evaluation of the BAA training with 262 Medi-Cal clinicians, an important step because a training could fail to affect practice or dissipate in impact over time. However, in our sample of clinicians, we found strong evidence that the training has had a lasting impact on clinicians' practice. When surveyed six to 12 months after taking the training, 92.4 percent of clinicians who took our follow-up survey indicated improved knowledge of TIHC, and 85.7 percent reported that they pay more attention to their patients' ACEs than they did before the training. Specific skills that improved following the training included ability to talk with patients about ACEs and build trusting relationships with patients, key elements of TIHC. Ninety percent of respondents to the follow-up survey indicated that they currently screen patients for ACEs, and 75.1 percent indicated that the training has had a positive impact on their patients. Although results from our relatively small follow-up sample might not capture the experiences of all Medi-Cal clinicians, these survey findings suggest that UCAAN should preserve and build on the success of the BAA training program.

The clinical impact of UCAAN's training efforts is most clearly demonstrated in the evidence for increases in ACE screening and referral in UCAAN projects and in Medi-Cal statewide. Medi-Cal claims demonstrate that 16.5 percent of Medi-Cal beneficiaries received at least one ACE screening during the first three years of implementation. However, these data also indicate that screening has been uneven. While 31.0 percent of Medi-Cal enrolled children have been screened, only 6.3 percent of Medi-Cal adults have been screened. The low rate of screening among adults relative to children highlights the challenge of expanding screening for the adult Medi-Cal population.

There Is Widespread Demand for UCAAN Training from Diverse Audiences

We found that not only have there been large numbers of participants in UCAAN trainings, many of whom are not clinicians, but there are also multiple motivations for participation and interest in additional trainings. Although some clinicians indicated that they took the training because their employer had asked them to, many had other motivations, such as increasing knowledge of ACEs and toxic stress and benefiting patients. While many of the clinicians trained were in primary care practices, one-third specialized in behavioral health, suggesting interest in how ACEs should be addressed in specialty behavioral health care. Satisfaction with the trainings was high at the time of the training, and follow-up interviews indicated sustained impacts of the BAA and supplemental online trainings that UCAAN provides on TIHC topics, for which providers can earn continuing education credits. There was broad interest in further training, despite some respondents citing lack of time or awareness of additional trainings on the ACEs Aware Learning Center, the online platform that houses all of the ACEs Aware training content (online courses and live webinars). Such “booster” trainings may be warranted; there was a significant increase in the percentage of clinicians listing “insufficient training” training as a barrier to ACE screening six to 12 months following the BAA. In addition to the content of the training, learners appreciated the ease of access to the training programs, the clarity and scientific grounding of the content, and the use of clinical cases to relate the knowledge to clinical practice. Specific topics of interest for future trainings included how to integrate ACE screening and TIHC with clinic workflows, including other screening tools that are commonly used in primary care, and techniques for talking with patients about their ACE screening results (including such challenging topics as experiences of sexual abuse).

UCAAN Has Made Some Early Progress in Improving Systemic Capacity to Provide TIHC

System-level change in health care systems can take years or even decades to achieve (Morris, Wooding, and Grant, 2011)—a time horizon well beyond the current evaluation period. UCAAN's work provides a strong foundation for this kind of change, but there are still opportunities for future growth. Systemic impacts to date include the large number of people who have taken the BAA training or supplemental trainings who are not clinicians but are involved in health care delivery; certification of BAA trainees for Medi-Cal reimbursement for ACE screenings, which contributes to financial sustainability of screening; the impacts of Medi-Cal clinicians as champions for ACE screening and TIHC in the places where they work; and the large number of ACE screenings that have been reimbursed by Medi-Cal.

The challenge of promoting system change is evident from the results of our six- to 12-month follow-up survey of Medi-Cal clinicians. Despite the fact that the respondents reported high levels of screening within their own practices, at follow-up they reported implementing fewer changes to support TIHC than they had initially intended to implement immediately after taking the BAA training. In addition, they were also more likely to report implementation barriers, such as system constraints and lack of referral resources, at follow-up than they were initially. Technical assistance (TA) and other support to clinics could be effective at addressing some of these barriers but would require additional development to be effective. We found that the responses to inquiries for TA were quick and opinions of its usefulness were generally positive, but only a small number of people took advantage of the service, even when resources to support clinic implementation were offered. The need for support in clinic-level implementation, which TA aims to provide, is clear from multiple requests for implementation-level guidance for ACE screening and TIHC.

Pilot Projects Have Built an Impressive Body of Knowledge About Implementation of ACE Screening and TIHC in Diverse Health Care Settings

UCAAN's pilot project program has focused on challenging aspects of implementing ACE screening and TIHC across the health system from high-intensity settings (such as inpatient pediatric care) to underserved populations in the community (such as migrant farm workers). As a group, the pilots demonstrate the enormous complexity of addressing ACEs and providing TIHC across the myriad clinical sites in the health care system and for the diverse population of Medi-Cal beneficiaries. Learnings from the pilot projects address many of the barriers to ACE screening that we identify in this study, and those learnings provide valuable knowledge that can inform the development of UCAAN training strategies and materials. The pilot project findings underline the importance of developing clear implementation models that can guide clinics through the process of implementation. They also indicate that one size will not fit all; guidance on implementation should be tailored to clinical settings, patient populations, and staff roles, including non-clinical staff, and should prioritize developing trusting relationships along with collection of screening data. The pilot projects have also contributed to understanding the impact of TIHC on quality of care, focusing attention on the overarching goal of building trusting relationships between patients and providers as a foundation for effective health care delivery.

Recommendations

Based on our findings, we suggest the following recommendations to strengthen UCAAN's impact in the future.

Expand the Reach of the BAA Training, and Develop Additional Courses to Support TIHC

UCAAN should build on the success of the BAA training by expanding the training's reach to Medi-Cal clinicians throughout the state and developing new trainings for diverse groups of clinical support staff. Outreach efforts should aim to reach parity across the state's regions in the proportion of Medi-Cal clinicians who have taken the training, with efforts to increase the number of trainees in the Inland Empire and Southern San Joaquin Valley regions. Outreach could be targeted to clinics that have not conducted screening and individual clinicians who have not taken the BAA course. To address the relatively low proportion of adults who have been screened, special attention should be given to clinics that serve adults and to adult primary care providers. The finding that most clinicians learned about trainings through the ACEs Aware website or from colleagues at their clinic suggests that word of mouth through provider networks could be a useful outreach strategy. In addition, promotion of training programs through managed care organizations (MCOs), which often send newsletters to providers, could also be effective in reaching providers who are not aware of ACE screening. Trainings for non-clinician staff, including administrative leaders and office staff, could improve clinic-level capacity and are consistent with principles of TIHC (Machtinger et al., 2015). Trainings should also strive to address other topics raised by interviewees: integration of ACE screening into clinic workflows, the relationship between ACE screening and other developmental and behavioral health screenings also being conducted, how to respond to high ACE scores in the primary care setting, contextualizing ACEs for patients and their families, and responding to disclosure of difficult experiences, such as childhood sexual abuse.

Translate Pilot Project Findings into Practical Implementation Guidance

The pilot projects have produced a wealth of experience and knowledge about implementation of ACE screening in a wide range of health care settings, but the findings need to be translated into practical implementation guidance to be useful to implementors. Some of the pilot projects have already produced toolkits based on their findings, but these have not drawn on integration of learnings across pilot projects. More detailed and comprehensive implementation guidance, informed by the pilot projects, would address the need for implementation assistance voiced by interviewees and people seeking TA. We recommend adopting a knowledge translation model, such as that described by Graham et al. (2006), to guide UCAAN's efforts to integrate findings from multiple pilot projects with clinic experiences and the research literature and disseminate the findings as actionable guidance to clinics. This effort, carried out by a group of experts that includes members of the pilot project teams, could focus initially on one type of clinic or population for which the pilot projects have produced robust results. The process could be conducted in collaboration with clinics implementing ACE screening that can provide feedback on the practical relevance of the materials. The knowledge translation model would then provide recommendations for dissemination of the implementation materials.

Improve Guidance and Support for ACE Screening and TIHC in Clinics Serving Adults

To address the low rate of screening among adults, implementation support materials should be developed to address the distinct challenges of implementing screening in clinics primarily serving adults, including family practice and adult primary care clinics. Clinics serving adults face different challenges than clinics serving children. The rules regarding reimbursement are less favorable; patterns of care are different, with fewer adults making regular primary care visits than children; patients' health and social needs differ; and ACEs are more distal for adults than for children, even if their impacts persist. Additional studies, including quantitative analysis of claims data and qualitative investigations of factors influencing screening of adults, could contribute to actionable implementation guidance for these clinics. The California Department of Health Care Services should consider changes to reimbursement policy that would support TIHC for adults beyond the initial once-in-lifetime screening. By current policy, individuals with high ACE scores qualify for enhanced care management. Support for less-intensive follow-up services, such as supplemental payments for outpatient visits, would have broader reach while directly supporting primary care clinics.

Develop Whole Clinic Training and Implementation Support Materials

To address the barriers experienced by trainees and requests for detailed guidance on implementation, UCAAN should review existing trainings on the ACEs Aware Learning Center (such as the Implementation with Intention training series) and potentially develop additional implementation support materials that provide “whole clinic” approaches to ACE screening and TIHC. These materials could draw extensively on the learnings from the pilot projects (building on current offerings in the Clinical Pilots training series), as well as other clinic experiences and health services research. The materials should cover not only implementation of ACE screening but also extensive planning related to TIHC more generally, including appropriate screening response and referral procedures. An initial step could focus on developing materials for one type of clinic, such as rural Federally Qualified Health Centers or small private pediatric practices. The materials could include web-based trainings with different tracks for clinical and administrative staff, direct consultations with UCAAN staff, peer-to-peer learning opportunities, input from local MCOs related to billing, and small grants to support planning and development. Subsequent work could focus on additional types of clinics.

Streamline Technical Assistance

The TA provided by UCAAN had many strengths, including a very quick response time, and was found to be helpful by recipients. However, there are opportunities to increase the efficiency and consistency of the TA and improve the impact on clinics. We divide these recommendations into two main categories: managing the volume of requests and standardizing TA responses.

Manage the Volume of Requests

The ACEs Aware TA team received a large number of inquiries that were not actually related to implementation, and these requests require time and effort to manage. Many requests for assistance were about licensing for the Pediatric ACEs and Related Life Events Screener (PEARLS) screening tool, and a small number included requests by individual patients who were seeking assistance for their personal trauma and mental health issues. These requests were quite different from those of people requesting implementation assistance. To focus efforts on implementation assistance, we recommend the following:

1. Clarify the language in the PEARLS screening tool to clearly explain the licensing requirements for paper-based and electronic health record–integrated screening to reduce the number of inquiries about use of the tool.
2. Develop procedures for rapid referral to assistance, such as the suicide hotline, for individuals who contact ACEs Aware seeking help with an immediate mental health problem.
3. Conduct outreach with organizations in California so that they are aware that they can request TA for ACE screening implementation. This may include collaborations with MCOs to divert certain types of questions their way, as well as increased advertising of the clinical implementation pilots.

Standardize Technical Assistance Responses

In the analysis of the TA email conversations, we found variations in how questions were handled and answered. The currently available data could be used to build standard responses to common questions that staff can use to prepopulate responses to questions they receive. Standardized responses should be developed to address some common themes identified in the analysis of the TA inquires, such as submitting claims for ACE screening and response, fitting ACE screening and response into clinical workflows, and general resources on TIHC—beyond ACE screening and response.

Improve Referral Networks

The lack of accessibility to services to which patients with high ACE scores can be referred is one of the most cited barriers to ACE screening and response. This is clearly a systemwide structural issue that UCAAN and its partners cannot fully address acting alone. However, there are steps they can take to address these barriers. First, trainings can emphasize the ability of primary care providers to effectively respond to patients with high ACE scores within the primary care context, through patient-provider communication about the implications of ACEs for health and health care. Improving provider confidence in responding to high ACE scores may reduce the need for referrals to specialty care. Second, implementation guidance and TA could provide clinics with efficient techniques for strengthening network connections with local behavioral health and social services providers and implementing systems for tracking referrals. Although it is unrealistic to expect clinics to have in-house behavioral health services, local providers could be better used as referral resources. Third, UCAAN should work with MCOs and the state to strengthen referral networks, giving clinics confidence that they have access to resources to address the needs of patients with high ACE scores. Navigation services, supported by evidence from the pilot projects, could play a role. MCOs are aligned with UCAAN interests in strengthening these networks, particularly under the California Advancing and Innovating Medi-Cal (CalAIM) reforms that aim to link health care providers with resources to address patients' health-related social needs. In prior work, UCAAN has supported referral network–building, through the Preventing and Responding to ACE-Associated Health Conditions and Toxic Stress in Clinics through Community Engagement (PRACTICE) community grant program (Chen et al., 2024) and the earlier “network of care” grants. Working with MCOs to identify community-based referral networks and provide direct consultations to clinics could leverage CalAIM resources and advance these goals on a statewide basis.

Logic Model

The impact evaluation was guided by the logic model shown in Figure 1. As shown in the model, the evaluation focused on three impact domains. First, clinical skills refers to the technical clinical skills of individual clinicians who provide direct care to Medi-Cal beneficiaries. Second, system capacity for TIHC refers to broader change within the health care delivery system, including within clinics, that enable and facilitate provision of ACE screening and response and TIHC. Third, ACE screening and response refers to screening for ACEs in Medi-Cal beneficiaries and providing appropriate clinical responses.

UCAAN activities are BAA training, supplemental TIHC trainings, technical assistance, UCSF pilot projects, UCLA pilot projects, the PRACTICE program, and ACEs-LA Network of Care.

Capacity for TIHC includes two items. Clinical skills: How has UCAAN improved TIHC knowledge and skills in the clinical workforce serving Medi-Cal beneficiaries? System capacity for TIHC: How has UCAAN improved trauma-informed systems of care serving Medi-Cal?

Delivery of TIHC has one item. ACE screening and response: How has UCAAN increased ACE screening and referrals for Medi-Cal beneficiaries?

Evaluation Approach

To assess UCAAN impacts across these domains, we followed a three-step process for analyzing each component. First, we reviewed information on UCAAN activities, which involved interviews with UCAAN staff to understand their organizations and goals, analysis of administrative data, and qualitative analysis of communications and other documentation. Second, where possible, we collected new data for the evaluation using web-based surveys and semi-structured qualitative interviews. Third, we summarized the findings to assess UCAAN impacts and identify recommendations to inform UCAAN's future work. Specific methods for each impact domain included the following.

Clinical skills: We analyzed participation and evaluation data on the BAA training program, which provides an introduction to ACE screening and response. Data come from more than 40,000 learners who completed the BAA and 12,802 Medi-Cal clinicians who completed a post-training course evaluation. We conducted a follow-up survey of 262 Medi-Cal clinicians who took the BAA training to assess its impact on their clinical practices six to 12 months after taking the training. The survey was supplemented by qualitative interviews with a subset of respondents (n = 14).

System capacity: We examined impacts of three UCAAN activities on system capacity: trainings (BAA training and supplemental trainings), TA, and pilot projects. Follow-up surveys (n = 92) and qualitative interviews (n = 4) were conducted with learners who completed supplemental trainings on the ACEs Aware Learning Center and recipients of TA. Reports submitted by the pilot project teams were analyzed to identify common themes and summarize general findings.

Screening and response: We drew on existing reports and Medi-Cal claims data to assess UCAAN's impact on screening and response. The reports include an evaluation of the PRACTICE community grant program (Chen et al., 2024) and an analysis of data on ACE screening and referral in Los Angeles County clinics conducted by UCAAN evaluators through an endeavor that they called their Innovation Lab. Data on paid claims for ACE screening and response were used to describe screening for Medi-Cal beneficiaries across the state.

Conclusion

In this evaluation, we found that UCAAN has had a major impact on the capacity of individual clinicians who treat Medi-Cal beneficiaries to provide TIHC, largely through the BAA training program. The BAA course has trained and continues to train a significant proportion of Medi-Cal primary care clinicians, and our survey results indicate that the impact of the training on care and on patients is lasting. UCAAN's impact on system-level change has understandably been more gradual. There have been positive impacts, most clearly demonstrated by the fact that clinics have been reimbursed for providing ACE screening and response to more than 2 million Medi-Cal beneficiaries. However, Medi-Cal clinicians who have been trained to provide ACE screening and response indicate that they continue to face barriers to providing TIHC in the clinics in which they work. UCAAN is making important contributions to addressing these barriers. Most importantly, the pilot programs have made major contributions to knowledge of how ACE screening and response and TIHC more generally can be implemented throughout the health care system. These findings provide a unique opportunity for translation into practical guidance that will support implementation in the future. The detailed findings contained in this study can help guide UCAAN through its next stages of development, taking stock of its successes and opportunities for further growth in planning future strategies.