In 2004, California voters passed the Mental Health Services Act. The Act was intended to transform California's community mental health system from a crisis-driven system to one that included a focus on prevention and wellness. The vision was that prevention and early intervention (PEI) services marked the first step in a continuum of services designed to identify early symptoms and prevent mental illness from becoming severe and disabling. Twenty percent of the Act's funding was dedicated to PEI services. The Act identified seven negative outcomes that PEI programs were intended to reduce: suicide, mental health-related incarcerations, school dropout, unemployment, prolonged suffering, homelessness, and removal of children from the home.
The Mental Health Services Oversight and Accountability Commission coordinated with the California Mental Health Services Authority (CalMHSA), an independent administrative and fiscal intergovernmental agency, to seek development of a statewide framework for evaluating and monitoring the short- and long-term impact of PEI funding on the population. CalMHSA selected the RAND Corporation to develop a framework for the statewide evaluation.
Interviewing Key Stakeholders
In order to develop the goals for the evaluation framework, RAND researchers conducted interviews with 48 key stakeholders and elicited their perspectives on how the frameworks might be used as well as their ideas for attributes that would make the frameworks useful.
We used a widely accepted model of how health services affect health to develop our overall framework and applied it to the specifics of PEI implementation.
We created two types of frameworks: an “overall approach” framework and specific frameworks for each of the key outcomes specified by the Act. The frameworks identify, at the conceptual level, the key components that should be measured and tracked over time, and they can provide information that would be useful to a broad range of stakeholders and decisionmakers (including state planners interested in the mental health of California's population), consumers and individual providers.
The frameworks include individual and family outcomes (population-level measures of emotional well-being and family functioning), program and service-system outcomes (the quality and timeliness of treatment and increased collaboration across agencies), and community outcomes (stronger and more resilient communities, as well as population-level measures of negative outcomes, such as unemployment or suicide).
Overall Approach Framework
Figure 1 depicts the overall approach framework for the evaluation. The framework asks a series of questions about PEI funding: Where is the funding going, what it is being used for, does the funding make a difference, and are there resulting public health benefits?
An Approach to Understanding the Impact of Prevention and Early Intervention Funding
Moving from left to right in the figure, we see the following:
- Box 1, “PEI Funding”: The initial community planning process in each county to determine funding priorities.
- Box 2, “Where is it going?”: The types of programs that were funded using PEI resources and the programmatic capacity that was developed.
- Box 3, “What is it doing?”: The “process” of delivering the programs—determining what prevention activities reached which target populations.
- Box 4, “Does the funding make a difference?”: The direct, short-term outcomes that PEI is intended to bring about—changed knowledge, behaviors, and attitudes, as well as improved resilience and emotional well-being—measured at the population level.
- Box 5, “Are there public health benefits?”: The ultimate outcomes measured at the population level. Changes in short-term outcomes are intended to reduce these seven negative outcomes identified by the Act.
In most cases, the data relevant to boxes 2 and 3 would be provided by programs and counties. Data relevant to boxes 4 and 5 would come from existing national or statewide surveys or vital statistics. The social and economic contexts influence how PEI was implemented and what it is accomplishing; therefore, socioeconomic context is shown at the bottom of the figure as affecting all of the components.
Examples of Outcome-Specific Frameworks
We developed an evaluation framework for each of the key outcomes identified by the Act.
Analytic Approaches to Evaluating the Impact of Prevention and Early Intervention
Inherent Limitations of a Prevention and Early Intervention Evaluation
A PEI evaluation has some important inherent limitations. Because the programs and activities were not randomly implemented and there are no geographic areas or populations within California that were not exposed to PEI activities, it would be technically difficult (although not impossible) to estimate the causal impact of PEI on outcomes. What can be done more easily is to relate changes in PEI program activity to changes in outcomes, without establishing causality. A second limitation is the fact that PEI programs and services were meant to function as part of a continuum of services that included treatment and recovery services. Unless some population groups were systematically exposed to one program but not the others, it is not analytically possible to separate the impact of PEI from those of other treatment and recovery services.
There are three evaluation designs that could be used to estimate the impact of PEI funding on outcomes:
Time-Trend Analysis of Observational Data (Before-After Design)
In this design, the evaluator compares outcomes for the study population before and after a program is implemented. This evaluation design is simple and often easy to implement, but it is also not as robust as other designs. The principal limitation is that it is difficult to distinguish the “causal” effect of the program from the effect of overall time trends.
This approach compares what happens in California with what happens in other states that are similar to California and assumes that time trends would be the same in the treated and comparison groups. If data were collected each year, it would be possible to document the yearly “benefit” of PEI program activity and to assess how utilization and outcomes are affected by changes in the social and economic context.
Synthetic Control Method
This method modifies the difference-in-differences (D-in-D) framework to make it particularly suitable for evaluating programs in which, like PEI, there is only one “treated” unit—in this case, California. This approach produces a much better comparison group than one in which all the untreated units are essentially given the same weight.
Using Descriptive Statistics for Inference
Our evaluation framework can also be used to monitor the effects of PEI programs by collecting and reporting descriptive information or statistics. Descriptive data can help policymakers to continuously monitor progress toward benchmarks and can serve as “early warning” indicators of implementation failures. An effective and efficient way to provide descriptive data about PEI programs is to create a web tool.
Usefulness of the Evaluation Framework
The negative outcomes identified by the Act are broad social outcomes that are affected by many different social forces, and changes in these outcomes will take years to observe. Although it is analytically possible to evaluate the causal impact of the Act on population-level outcomes, we do not recommend this approach. Rather, we suggest using existing data to track over time the population-level outcomes identified in the Act and ultimately to provide the data needed to estimate how this historic initiative has affected the mental health of California's population. This is an excellent time to establish a surveillance system that can be used to provide important information about the early phase of PEI activity. We recommend using resilience and emotional well-being to monitor and track changes at the population level.
We recommend additional data development to support implementation of the evaluation framework:
Immediate Prevention and Early Intervention Program Information Needs
It is essential to develop standardized, core information about the programs funded under the Act's PEI initiatives, the activities carried out by these programs, and the individuals reached by these activities. At minimum, all programs should report on the number of individuals served or exposed to the intervention, the type of program, and the target population. A next step would be for programs to report on the demographic and social characteristics of the individuals they reach. The last (and significantly more difficult) step would be to implement data systems that can track individuals across programs and service systems.
Prevention and Early Intervention Performance Indicators
Currently, there are few standardized and widely accepted measures of the quality of PEI services, but measures could be developed over time. Some examples of potential performance indicators include: whether a program meets certification standards, client satisfaction with program activities, and whether training or other interventional activities are delivered with fidelity to evidence-based protocols.
Maintaining and Improving Tracking of Population Outcomes
Existing data sources can be used to populate constructs in the PEI evaluation framework, but in some cases, these data sources could be improved. A key example is suicide statistics. National standards provide guidelines for more-consistent reporting, and these could be adopted to improve suicide statistics and their utility for PEI evaluation.
Other Important Evaluation Issues
Evaluating Program Efficacy
In many cases, the literature provides insufficient evidence on the efficacy of specific PEI activities. We recommend that the state or counties strategically develop the evidence base for PEI programs by conducting rigorous evaluations of strategically selected promising programs.
Evaluating Cultural Competence
There are currently no broadly accepted and reliable measures of cultural competence that could serve as performance indicators in an ongoing statewide monitoring system. If the development of cultural-competence assessments at the program level is a priority, we recommend obtaining advice from national experts.
Developing Program Capacity for Quality Improvement
Although routinely assessed outcomes are not useful to evaluate the comparative effectiveness of programs, they can and should be used for ongoing quality improvement efforts. We recommend developing program capacity for quality improvement.
We suggest a three-year phased implementation of the statewide evaluation framework.
The first year would include (1) demonstration of development and reporting of PEI program-level information; (2) psychometric assessment and refinement of program-level and population-level measures, which would also include pilot testing new measures; (3) development of descriptive analytic and reporting templates; and (4) proposed work plan and resources required for full implementation and ongoing maintenance. The second and third years would focus on implementing the full evaluation framework, including the infrastructure required to acquire, store, analyze, and routinely report data.
The research described in this report was sponsored by the California Mental Health Services Authority and was conducted within RAND Health, a unit of the RAND Corporation.