CalMHSA: Evaluation of California's Statewide Mental Health Prevention and Early Intervention Initiatives
May 8, 2014
The California Mental Health Services Authority (CalMHSA) — a coalition of California counties — has implemented an ambitious, first-of-its-kind set of statewide prevention and early intervention (PEI) initiatives to help reduce mental health stigma and discrimination, prevent suicide, and improve student mental health. Under the initiatives, program partners developed resources and programs and implemented PEI activities as part of a greater strategic plan aimed at improving mental health among residents. The plan was carefully developed through a process that involved diverse stakeholders (Clark et al., 2013). The programming was funded by Proposition 63, the Mental Health Services Act, and implementation began in 2011.
RAND's evaluation of CalMHSA PEI initiatives to date shows that program partners developed the capacities to deliver numerous new PEI activities and that their efforts have resulted in new infrastructure, staff expertise, resources, collaborative relationships, training programs, and educational materials developed and tailored for diverse target audiences. The statewide mental health PEI programming has reached large numbers of Californians and improved the knowledge, attitudes, and intentions of those who were reached.
This has all been accomplished in a relatively short time — four years to date — despite the many challenges inherent in developing new program activities on a statewide level. However, without sustained investment in programming, many of the gains will be lost. For example, the impacts of social marketing campaigns can be short-lived unless the campaigns are continued (Collins et al., 2012). Fortunately, the costs for sustaining already-developed interventions are much lower than the costs of initially developing them.
This brief reviews some of RAND's key evaluation findings regarding program implementation and short-term outcomes and discusses why sustained investment in programming and evaluation is important.
CalMHSA mental health PEI programs have used various mechanisms to reach a large number of Californians. These include the following:
Social marketing campaigns were associated with reduced mental illness stigma and greater confidence to intervene with those at risk for suicide. Social marketing campaigns aim to use concepts from marketing to change people's knowledge, attitudes, and behavior. The "Each Mind Matters" stigma reduction campaign has reached 1 in 4 California adults, and reach is growing (Burnam et al., 2014; Collins et al., 2015). Since the campaign's inception, the stigma of mental illness has decreased in California: More Californians say they are willing to socialize with, live next door to, and work closely with people experiencing mental illness, and they report that they are providing greater social support to those with mental illness (Collins et al., 2015). These changes are small but significant and are comparable to those attained by campaigns in other countries.
The suicide prevention social marketing campaign "Know the Signs" targets helpers of those at risk of suicide, and this campaign is also reaching a large number of Californians and having a positive impact. Experts in suicide prevention and social marketing campaigns have found that the campaign is aligned with best practices and hold it in high regard (Acosta and Ramchand, 2014). The campaign has reached more than half of Californian adults (Ramchand et al., 2015), and those exposed to the campaign report greater increases in confidence to intervene with those at risk for suicide than those who were not exposed to the campaign (Ramchand et al., 2015).
Trainings improved knowledge, attitudes, and beliefs about mental illness, and made participants feel more prepared to help those in distress. CalMHSA-funded trainings have also reached large numbers of people across the state, and the individuals who attend these trainings are benefiting from them, with participants experiencing small but significant changes in knowledge, attitudes, and intentions. For instance, trainings of staff in California's K–12 schools and of faculty, staff, and students in California's higher education systems were attended by large, racially and ethnically diverse audiences. Attendees not only are satisfied with the trainings but also report improvements in their confidence to refer and intervene with students who appeared to be emotionally distressed (Osilla et al., 2015a; Osilla et al., 2015b).
Several statewide programs have provided training that aimed to reduce stigma and discrimination toward individuals with mental illness. For instance, trainings conducted by Mental Health America of California and Disability Rights California, although different in nature, both successfully addressed a variety of stigma-related attitudes, beliefs, and intentions among attendees, with the largest changes in participants' willingness to interact with people experiencing mental illness (Cerully et al., 2015a). The National Alliance on Mental Illness (NAMI) successfully implemented more than 2,000 CalMHSA-funded trainings that reached over 18,000 individuals and had positive effects (Wong et al., forthcoming[a]; Wong et al., forthcoming[b]). NAMI's "Ending the Silence" training for high school students resulted in immediate and substantial improvements in students' knowledge about mental health, as well as in positive shifts in emotional responses and attitudes toward people with mental health challenges. NAMI's adult educational programs — In Our Own Voice (IOOV), Parents and Teachers as Allies (PTasA), and Provider Education Program (PEP) — also improved adults' attitudes toward persons with mental illness, particularly in their willingness to socialize with, live next door to, and work closely with individuals with mental illness.
CalMHSA-funded suicide prevention trainings also appear to be a good investment. CalMHSA has funded a cadre of Californians to become trainers in Applied Suicide Intervention Skills Training (ASIST). RAND observed a few training sessions and found that these new trainers demonstrated high fidelity to the prescribed training (Osilla et al., 2014). Cost-benefit analyses of ASIST showed that just one year of CalMHSA's investment in creating this group of ASIST trainers is projected to prevent at least 3,600 suicide attempts and 140 deaths over the next 28 years. By averting these suicide attempts and deaths, the state government will gain $50 for each $1 invested in ASIST through averted Medi-Cal health care costs and increased state income tax revenue (Ashwood et al., 2015a, 2015b).
Online resources reached many Californians, and efforts to drive people to websites were key. CalMHSA funded a number of websites and online resources intended to reduce mental illness stigma that have been visited many times — more than 650,000 visits in a year's time and more than 66,000 downloaded resources (Cerully et al., 2015b). The majority of website visits — 60 percent — originated from within California. However, more than 80 percent of California visits originated from the three largest metro areas — Los Angeles, Sacramento, and San Francisco — indicating a continued need to reach less populous areas of the state. Some program partners were more successful than others in encouraging Californians to use their websites, indicating that website sponsors need to invest in actively driving their target audiences to the resources in order for them to be used as designed. Thoughtful dissemination of online resources appears to be at least as important as their initial development and can leverage earlier investments.
Collaborative partnerships influenced mental health–related policies in California. CalMHSA supported program partners' efforts to develop collaborations to better serve communities, and our evaluation found that some collaborative groups have been able to influence policy or create and disseminate products with widespread impact across institutions and communities. For instance, a student mental health policy workgroup influenced state teacher credentialing standards: The workgroup recommended enhancing collaborative partnerships to link students to appropriate services, as well as requirements that K–12 educators be trained to facilitate students and their families' access to mental health services, and these standards were adopted by the state (Woodbridge et al., 2014).
Although CalMHSA's stigma reduction efforts are working, there is still much work to do in this area. Among people who themselves have mental illness, 9 out of 10 reported having been discriminated against in the past year because of their mental health problem; consistent with their perceptions of stigma and discrimination, more than 2 in 3 with mental illness would definitely or probably hide a mental health problem from coworkers or classmates, and more than 1 in 3 would hide it from family or friends (Wong et al., 2015). These high levels of perceived stigma and discrimination may help explain why almost half of California adults with serious psychological distress did not seek treatment in the past year (Wong et al., 2015).
One area for improvement is reaching subpopulations in California in which stigma toward those with mental illness is still high — for example, among Asian Americans, who make up 14 percent of California's population (Collins et al., 2014). There is also a need for targeted efforts to reach California's college campuses, where almost 1 in 5 higher education students reported probable psychological distress and high rates of impairment due to mental health problems (Sontag-Padilla et al., 2014). The need for intervention appears to be particularly high at California Community Colleges (CCCs): Despite having comparable levels of mental health issues, CCC students reported greater levels of mental health–related impairment yet lower rates of using or receiving referrals for counseling or mental health services, compared with students in the University of California and California State University systems. RAND's evaluation found that most higher education staff and faculty are unsure of the warning signs of mental health distress and do not have the skills to discuss mental health issues with students, and many remain uncertain about where to refer students with mental health needs. Most had not yet participated in trainings intended to foster such knowledge and skills. The largest barrier to participation was lack of knowledge that the trainings were offered (Sontag-Padilla et al., 2014), which underscores the importance of continued outreach to provide trainings shown to be effective (Osilla et al., 2015a).
These results are available because of CalMHSA's commitment to independent evaluation, including statewide monitoring of key mental health indicators. From a public health perspective, ongoing measurement of the mental health of Californians is critical for tracking needs, assessing longer-term impacts of PEI investments, and targeting future mental health PEI programming. CalMHSA's investments so far have provided a baseline (and relatively short-term follow-ups), and continued investment would make it possible to understand how key mental health indicators change over time.
Continued monitoring is especially important in California because the various components of the statewide mental health PEI efforts are designed to support and enhance one another to create greater change, and this reciprocal influence is likely to be evident only at a later time. Improving the mental health of an entire state will not happen overnight. Statewide monitoring efforts can make it possible to track progress and direct efforts where they are most needed.
CalMHSA's efforts to implement statewide mental health PEI programs to reduce mental health stigma and discrimination, prevent suicide, and improve student mental health have resulted in substantial progress over the past four years. RAND's evaluation provides evidence that PEI initiatives successfully developed resources and programs; reached large numbers of Californians; and improved the knowledge, attitudes, and intentions of those who were reached. However, the researchers were able to evaluate only relatively short-term, proximal outcomes, and most of the effects found so far were small in size — which is to be expected, given the length of time the programs have been operating. As a comparison, successful stigma campaigns in England and Australia have been under way for more than a decade (TNS BMRB, 2015; Wyllie and Lauder, 2012). Statewide PEI programming and evaluation efforts should be sustained to enable observation of cumulative effects on longer-term outcomes. Without strategic and continued investment in statewide PEI programming, many of the gains associated with the larger initial investment are at risk of being lost.
Acosta, Joie D., and Rajeev Ramchand, "Know the Signs" Suicide Prevention Media Campaign Is Aligned with Best Practices and Highly Regarded by Experts, Santa Monica, Calif.: RAND Corporation, RR-818-CMHSA, 2014.
Ashwood, J. Scott, Brian Briscombe, Rajeev Ramchand, Elizabeth May, and M. Audrey Burnam, Analysis of the Benefits and Costs of CalMHSA's Investment in Applied Suicide Intervention Skills Training (ASIST), Santa Monica, Calif.: RAND Corporation, RR-1115-CMHSA, 2015a.
Ashwood, J. Scott, Brian Briscombe, Rajeev Ramchand, Elizabeth May, and M. Audrey Burnam, "Should California Continue to Invest in Applied Suicide Intervention Skills Training (ASIST)? ASIST Could Save Lives and Wages and Reduce Medical Costs," Santa Monica, Calif.: RAND Corporation, RB-9849-CMHSA, 2015b.
Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, "What Has the CalMHSA Statewide Mental Health Prevention and Early Intervention Program Done So Far? Key Results from the Baseline RAND General Population Survey," Santa Monica, Calif.: RAND Corporation, RB-9771-CMHSA, 2014.
Cerully, Jennifer L., Rebecca L. Collins, Eunice C. Wong, Elizabeth Roth, Joyce Marks, and Jennifer Yu, Effects of Stigma and Discrimination Reduction Trainings Conducted Under the California Mental Health Services Authority: An Evaluation of Disability Rights California and Mental Health America of California Trainings, Santa Monica, Calif.: RAND Corporation, RR-1073-CMHSA, 2015a.
Cerully, Jennifer L., Courtney Ann Kase, Rebecca L. Collins, Eunice C. Wong, and Jennifer Yu, CalMHSA Stigma and Discrimination Reduction Online Resources: Highlights from an Evaluation of Web Analytic Data, Santa Monica, Calif.: RAND Corporation, RR-1237-CMHSA, 2015b.
Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, and Elizabeth Roth, Racial and Ethnic Differences in Mental Illness Stigma in California, Santa Monica, Calif.: RAND Corporation, RR-684-CMHSA, 2014.
Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, Dana Schultz, and Nicole K. Eberhart, Interventions to Reduce Mental Health Stigma and Discrimination: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative , Santa Monica, Calif.: RAND Corporation, TR-1318-CMHSA, 2012.
Collins, Rebecca L., Eunice C. Wong, Elizabeth Roth, Jennifer L. Cerully, and Joyce Marks, Changes in Mental Illness Stigma in California During the Statewide Stigma and Discrimination Reduction Initiative, Santa Monica, Calif.: RAND Corporation, RR-1139-CMHSA, 2015.
Osilla, Karen Chan, Dionne Barnes-Proby, Mary Lou Gilbert, and Rajeev Ramchand, A Case Study Evaluating the Fidelity of Suicide Prevention Workshops in California, Santa Monica, Calif.: RAND Corporation, RR-755-CMHSA, 2014.
Osilla, Karen Chan, Asha Goldweber, Rachana Seelam, Courtney Ann Kase, Elizabeth Roth, and Bradley D. Stein, Mental Health Trainings in California's K–12 System Are Associated with Increased Confidence and Likelihood to Intervene with and Refer Students, Santa Monica, Calif.: RAND Corporation, RR-953-CMHSA, 2015a.
Osilla, Karen Chan, Michelle W. Woodbridge, Rachana Seelam, Courtney Ann Kase, Elizabeth Roth, and Bradley D. Stein, Mental Health Trainings in California's Higher Education System Are Associated with Increased Confidence and Likelihood to Intervene with and Refer Students, Santa Monica, Calif.: RAND Corporation, RR-954-CMHSA, 2015b.
Ramchand, Rajeev, Elizabeth Roth, Joie D. Acosta, and Nicole K. Eberhart, Adults Newly Exposed to "Know the Signs" Campaign Report Greater Gains in Confidence to Intervene with Those Who Might Be at Risk for Suicide Than Those Unexposed to the Campaign, Santa Monica, Calif.: RAND Corporation, RR-1134-CMHSA, 2015.
Sontag-Padilla, Lisa, Elizabeth Roth, Michelle W. Woodbridge, Courtney Ann Kase, Karen Chan Osilla, Elizabeth D'Amico, Lisa H. Jaycox, and Bradley D. Stein, CalMHSA Student Mental Health Campus-Wide Survey: 2013 Summary Report, Santa Monica, Calif.: RAND Corporation, RR-685-CMHSA, 2014.
Wong, Eunice C., Rebecca L. Collins, Jennifer L. Cerully, Elizabeth Roth, and Joyce Marks, Stigma, Discrimination, and Well-Being Among California Adults Experiencing Mental Health Challenges, Santa Monica, Calif.: RAND Corporation, RR-1074-CMHSA, 2015.
Wong, Eunice C., Rebecca L. Collins, Jennifer L. Cerully, Elizabeth Roth, Joyce Marks, and Jennifer Yu, Effects of Stigma and Discrimination Reduction Trainings Conducted Under the California Mental Health Services Authority: An Evaluation of NAMI's Ending the Silence, Santa Monica, Calif.: RAND Corporation, RR-1240-CMHSA.
Wong, Eunice C., Rebecca L. Collins, Jennifer L. Cerully, Elizabeth Roth, Joyce Marks, and Jennifer Yu, Effects of Stigma and Discrimination Reduction Trainings Conducted Under the California Mental Health Services Authority: An Evaluation of the National Alliance on Mental Illness Adult Programs, Santa Monica, Calif.: RAND Corporation, RR-1247-CMHSA, forthcoming(b).
Woodbridge, Michelle W., Asha Goldweber, Jennifer Yu, Shari Golan, and Bradley D. Stein, California K–12 Schools and Communities Collaborate to Support Student Mental Health, Santa Monica, Calif.: RAND Corporation, RR-688-CMHSA, 2014.
Clark, Wayne, Stephanie N. Welch, Sandra H. Berry, Ann M. Collentine, Rebecca Collins, Dorthy Lebron, and Amy L. Shearer, "California's Historic Effort to Reduce the Stigma of Mental Illness: The Mental Health Services Act," American Journal of Public Health, Vol. 103, No. 5, May 2013, pp. 786–794.
TNS BMRB, Attitudes to Mental Illness: 2014 Research Report: Prepared for Time to Change, April 2015.
Wyllie, Allan, and James Lauder, Impacts of National Media Campaign to Counter Stigma and Discrimination Associated with Mental Illness: Survey 12: Response to Fifth Phase of Campaign, Auckland, New Zealand: Phoenix Research, June 2012.
The California Mental Health Services Authority (CalMHSA) is an organization of county governments working to improve mental health outcomes for individuals, families, and communities. Prevention and early intervention programs implemented by CalMHSA are funded by counties through the voter-approved Mental Health Services Act (Prop. 63). Prop. 63 provides the funding and framework needed to expand mental health services to previously underserved populations and all of California's diverse communities.