The California Mental Health Services Authority (CalMHSA)—a coalition of California counties designed to provide economic and administrative support to mental health service delivery—formed the statewide Prevention and Early Intervention (PEI) Implementation Program. CalMHSA's focus in this effort is on three strategic initiatives: (1) Stigma and discrimination reduction, (2) suicide prevention, and (3) student mental health. Under each initiative, community agencies serve as PEI program partners, performing activities to meet the initiative's goals. In 2011, the RAND Corporation was asked to design and implement an evaluation of the three initiatives.
At CalMHSA's request, the RAND evaluation team is providing preliminary advice about which PEI activities seem most valuable to sustain or, in some cases, to enhance.
Criteria for Value in Sustainability
The RAND team developed and applied the following criteria in determining whether there was value in sustaining an activity:
- Does the activity fill a strategic gap? That is, is the activity an important intervention that is not redundant with county-level efforts?
- Does the activity employ population-based strategies? Strategies that can have a larger social impact through broad reach across a population can often only be implemented at the statewide level, given logistics and costs (e.g., social marketing campaigns). Thus, they should be a priority for state funding.
- Are there efficiencies in scale to be gained by statewide sustainment? Capacities for promoting and disseminating best practices that are more centralized (e.g., developing informational resources, training trainers in evidence-based interventions, or developing tools for intervention fidelity monitoring) can in some cases be more cost-efficient if implemented statewide rather than county by county.
- Would a relatively low-cost investment in sustaining program operations yield value and leverage already-created capacities that required a much higher start-up cost investment? For example, a website may be costly to develop, but once in place, it is relatively low cost to sustain.
- Would sustained programs have performance monitoring/improvement capacity integrated into them, so that interventions can be refined/improved over time?
- Would statewide efforts contribute to the development of valued local capacity (e.g., use of informational or evaluation resources by county-level mental health authorities, or by local community organizations or schools)?
Suggested Program Activities to Sustain
Our recommendations for program activities that we believe should be considered for sustainment are based on the criteria outlined above, literature reviews we conducted in each initiative area (Acosta et al., 2012; Collins et al., 2012; Stein et al., 2012), our knowledge of CalMHSA program partner activities gained through meeting with them and providing them with technical assistance, and our evaluation results to date.
Stigma and Discrimination Reduction Initiative
Social Marketing Campaign
We suggest that social marketing campaigns be sustained, as evidence shows that campaign impacts are short-lived unless continued (Collins et al., 2012). However, we generally recommend narrowing the focus to stigma (e.g., rather than forums and mutual support) and strategic dissemination of the Each Mind Matters (EMM) campaign.
Based on recent evaluation results, we believe that both the EMM campaign and the associated website show promise in reaching California adults, and should be sustained. Specifically, a RAND survey of California adults found a modest level of exposure to EMM early in the campaign; 11 percent of those surveyed said they had seen or heard the catch phrase “Each Mind Matters,” the new (at that time) slogan for CalMHSA efforts.
Although less than 1 percent reported visiting the EMM website, at the time of the survey it was just beginning to serve as the hub for CalMHSA dissemination and was primarily the host of the documentary “A New State of Mind” (Burnam et al., 2014).
Upcoming evaluation work. The RAND team is currently collecting follow-up survey data that will provide estimates of the degree to which exposure to the social marketing campaign has changed over time. Thus, more information on the campaign's reach will be available in the coming months. Future results of message-testing experiments (to be conducted in the fall of 2014) will also indicate which messages are most effective and thus most important to continue disseminating.
Results of the baseline survey also indicate that it will be important for CalMHSA to continue to target PEI efforts toward the ethnic and racial subgroups of Asian Americans and Latinos. This is applicable to social marketing campaigns as well as trainings and educational presentations, discussed further below. Asian Americans reported relatively high levels of stigma, as did Latinos who were interviewed in Spanish (Collins et al., 2014). Continuing to use the EMM website (both English and Spanish content) and other means to reach these groups is vital to overall campaign effectiveness.
Upcoming evaluation work. Analysis of the statewide follow-up survey will tell us the extent to which EMM is improving reach to these groups. RAND's future message-testing experiments will help to determine the effectiveness of EMM messages for these groups, including culturally targeted messages, and help CalMHSA identify the messages that work best with various groups.
CalMHSA has a particular interest in reaching young adults with the EMM campaign. RAND's preliminary analysis of the responses of 18–29 year olds to the baseline statewide survey suggests that young adults hold some of the least-stigmatizing attitudes toward mental illness. They are also the most likely to report contact with someone with mental illness in the past year. At the same time, however, fewer of them say they provided linkage to resources like treatment. Given their attitudes, it seems likely this lack of support is due more to a lack of knowledge of how to help than to stigma.
Based on these results, CalMHSA might play a role in addressing this knowledge gap by shifting the focus of the messaging for this particular group. Specifically, EMM messaging targeting young adults should focus on how they can provide support to individuals with mental health concerns and link them to resources, with less emphasis on stigma reduction.
Training in Stigma Reduction Interventions Delivered In Person to Target Audiences
We suggest sustaining trainings that are being implemented by program partner National Alliance on Mental Illness (NAMI), prioritizing those that focus on reaching “influencers” (i.e., corrections staff, employers, landlords, teachers, etc.). NAMI has advantages of both state- and local-level organizational structures that can sustain a workforce of presenters and facilitate access to key audiences over a longer term. NAMI also has a commitment to structured protocols and evaluation that should be directed to assessing fidelity and evaluating impact on an ongoing basis.
Among NAMI programs, In Our Own Voice (IOOV) should be prioritized because there is some evidence supporting its effectiveness and it can be used with multiple audiences. Further, recent RAND findings of greater stigma among some demographic groups (Collins et al., 2014), described above, underscore the importance of developing and sustaining a culturally and linguistically adapted training similar to IOOV that can be disseminated to Asian-American Californians and Spanish-speaking California Latinos.
Program partner United Advocates for Children and Families (UACF) has been facilitating administration of Mental Health First Aid (MHFA) training for counties. Because MHFA is also evidence-based, continued sustainment may be warranted. However, it is uncertain yet whether UACF has the capacity to implement and reach appropriate targets for MHFA trainings. As noted previously, CalMHSA has a particular interest in reaching young adults, and preliminary survey results suggest that young adults might benefit from learning how to provide support to individuals with mental health concerns. Given that MHFA addresses support-provision issues, it may be appropriate for MHFA sustainment efforts to target young adults in particular.
Upcoming evaluation work. RAND's forthcoming memo reviewing the evidence for MHFA effectiveness for various subgroups will help to inform decisions about the populations that should be targeted with MHFA.
Suicide Prevention Initiative
Social Marketing Campaign
We suggest continued dissemination of the “Know the Signs” (KTS) suicide prevention campaign to sustain its effects. RAND's survey data found a relatively high level of exposure to KTS campaign materials. Moreover, those who reported seeing KTS campaign materials report being more confident in intervening with those at risk of suicide (Acosta et al., 2014). Further, an expert panel convened by RAND came to a consensus that the KTS campaign exceeded most other social marketing campaigns, although there are still some areas that could be improved (data analysis is ongoing, and additional results will be available in the fall of 2014). Exposure to KTS messages was significantly lower among Asian Americans than other groups (Ramchand and Roth, 2014), and Latinos and Asians who took the survey in a language other than English were less likely to report being exposed to the KTS campaign.
Based on these early findings, we recommend continued dissemination of the KTS campaign and provision of technical assistance to counties. Counties can assume responsibility for identifying and publicizing materials in targeted venues. We further suggest that it may be helpful to target the campaign toward specific groups. Given our finding of lower campaign exposure among some racial and language groups, we recommend that outreach efforts include culturally and linguistically appropriate campaign dissemination to reach racial/ethnic minorities, particularly California's Asian-American community. The campaign may also have a larger impact if targeted to gatekeepers (e.g., teachers, youth coaches, senior centers, retirement and nursing home staff) rather than the general population because they may have greater access to those who need help and may be more likely to already be in a helping role.
Finally, given research evidence that “means restriction” (i.e., making the means by which individuals kill themselves less readily available) is a highly effective strategy for preventing suicide (see Acosta et al., 2012), the RAND team recommends that suicide prevention messaging include an additional focus on means restriction to fill an important gap in current efforts. This might include messaging about bridge safeguards, proper gun storage and handling, promotion of drug buyback programs, etc.
Upcoming evaluation work. RAND is conducting a follow-up survey (summer of 2014) that will enable, among other goals, tracking further exposure to the KTS campaign. Message-testing experiments will also be conducted (in the fall of 2014) that can inform the development of effective messages going forward.
Training in Suicide Intervention Skills Delivered In Person to Target Audiences
We suggest that Applied Suicide Intervention Skills Training (ASIST) become more focused on vocations that are likely be in a position to provide help (e.g., police, school counselors). RAND's fidelity and adherence evaluation (currently under review) found that trainers need to improve their competencies at tailoring training content to be more relevant to specific audiences. Thus, for trainings targeted for specific vocations, an emphasis should be placed on ensuring that group-specific trainings are conducted by experienced trainers familiar with target audiences and that all trainers are taught about appropriate adaptations and tailoring strategies for specific audiences.
RAND developed a fidelity and adherence protocol to monitor the quality and delivery of ASIST trainings. Ongoing use of this protocol by mentors is recommended so that trainers can receive structured and guided feedback on areas for improvement.
To fill a key gap in PEI programming, the RAND team recommends a new focus on training primary care providers and health care organizations in depression identification and integrated care delivery, given research evidence that improved depression awareness among health care professionals can lead to reductions in suicides (see Acosta et al., 2012, for a review).
Hotlines/Warmlines Operated by Regional/Local Crisis Centers
Another recommendation is to continue efforts to evaluate and improve the quality of hotlines/warmlines, including initiating regular reporting of common metrics on calls, pursuing accreditation, and monitoring quality of calls. RAND has developed a call-monitoring protocol and has conducted live call monitoring at ten crisis centers. In addition to reporting on the overall results of this evaluation in the fall of 2014, RAND will prepare individual reports for those ten centers using their results, and will provide technical assistance to help them use the results to gauge and improve quality.
We suggest that crisis centers continue to use live call monitoring as a tool for continuous quality improvement. Crisis centers should be encouraged to identify long-term sustainability plans and identify whether community-based partnerships can be leveraged to establish linkages to support sustainment.
The current crisis center system has many local call centers with many different numbers. It is unclear whether the current reliance on many call centers is inefficient and, more broadly, whether the benefits of a local approach to crisis intervention outweigh the model's disadvantages. Note that promoting local numbers without long-term sustained funding is potentially risky, as some numbers may not be reliably available when needed. It is important for the state and local regions to consider the pros and cons of this local model as compared to alternative call center models, including blended call centers (e.g., centers that also receive “211” calls) or a more centralized system.
Upcoming evaluation work. RAND is currently investigating the key issue of whether it is worthwhile to sustain many call centers versus moving to a more centralized system. The RAND team plans to produce an infographic this summer that lays out the trade-offs of different approaches (i.e., pros and cons of the two alternative delivery systems), to be followed by a longer paper this winter.
In-Person and Online Suicide Prevention Services
It may also be helpful to broaden suicide prevention strategies beyond hotlines and other currently funded activities. RAND has found that people would prefer face-to-face help for suicidal thoughts, although this preference is not as strong in some minority groups, who are more interested in text and web-based chat services as compared to Whites (Ramchand and Roth, 2014). Therefore, we recommend sustained support of a diverse array of crisis services, ranging from in-person services to online services.
Strategy to Reach the Elderly at Risk of Suicide
It is well documented that suicide rates are elevated among the elderly, especially older males (Acosta et al., 2012). Although the KTS campaign is disseminated to broad populations in an effort to reach helpers of older adults, the campaign does not directly target elderly individuals at risk of suicide. Based on the research literature (see Acosta et al., 2012), the RAND team recommends that CalMHSA develop and implement a plan to address the high risk of suicide among the elderly, including directly targeting at-risk elderly individuals with campaign messages.
Intersection with Student Mental Health
Some suicide prevention initiative programming may be most effectively administered in school-based settings. Evidence-based programs like the Signs of Suicide Prevention Program (SOS) employ a combination of gatekeeper training and school-based screening to identify students at risk of suicide. We believe it would be valuable to facilitate the adoption of these programs into schools to identify and intervene with youth at risk of suicide.
Student Mental Health Initiative
Training School Faculty/Staff/Students in Evidence-Based Interventions
Since a substantial amount of training is being conducted in school systems, a more focused approach to sustainment may be needed. To maintain capacity in a cost-effective manner, it might be possible to invest in a sustainable training capacity (for instance, through online learning) for the most useful or in-demand trainings. Examples include the Kognito Suicide Prevention training—adopted systemwide by the California Community Colleges (CCC)—which is available online to students, staff, and faculty by various special topics, such as veterans, LGBTQ, etc. We also emphasize sustaining those training activities that have a greater evidence base (e.g., ASIST, MHFA).
Our Higher Education Campus-Wide Survey results (Sontag-Padilla et al., 2014) indicate that less than 40 percent of faculty/staff believe they have the skills to directly help students with mental health problems. Yet, on average, only a subset of staff/faculty (12–29 percent; min = 8 percent, max = 57 percent) participated in student mental health (SMH)–related training in the six months prior to the survey. The largest barrier was not knowing that trainings were offered.
Higher education systems and campuses with higher rates of training participation should share their success with engaging staff/faculty to attend trainings. We also recommend that resources be spent on broadening visibility of trainings for staff/faculty and providing incentives for their attendance.
Because the SMH websites are relatively inexpensive to maintain, and our evaluation suggests increasing use of these websites, we recommend efforts to sustain these web-based resources, as well as continued efforts to publicize them. It may also be worthwhile to consider efforts to “cross fertilize” multiple program partner websites with material that is drawing the most views.
Specifically, data from RAND's evaluation of the K–12 and higher education websites demonstrates that, since their launch, program partners have experienced increased use of their websites. From October 2012 to March 2014, the California County Superintendents Educational Services Association's (CCSESA's) Regional K–12 SMH Initiative website (http://www.regionalk12smhi.org/) received 11,573 visits (37,165 page views) to their website clearinghouse. For higher education partners, California Community College's Student Mental Health Program (SMHP) website (http://cccstudentmentalhealth.org) received 22,690 visits (54,462 page views) from April 2013 to March 2014 and the University of California (UC) Student Mental Health Resources and Promising Practices website (http://www.ucop.edu/student-mental-health-resources) received 2,033 visits (7,128 page views).
Over 90 percent of visits to the systems' websites originated from within California, suggesting that the targeted campaign to K–12 and higher education stakeholders in California was successful at engaging the intended audience. Additionally, the use of search engines to access the website increased across the reporting periods for CCSESA and UC, suggesting an increase in awareness of the website and the use of search terms related to site content. Promotional campaigns have also successfully increased website traffic, with one CCC campaign resulting in a 600-percent spike in direct traffic (e.g., users typing URLs into their browsers, accessing bookmarks, or clicking links in emails).
Finally, from our resource topic analysis of CCSESA's website clearinghouse, anger management was consistently the most viewed of the resource topics (24–29 percent of page views each quarter), followed by mental health/wellness, bullying, behavior management, and suicide. Within the most-viewed topic, programs and practices was the most viewed content type (57 percent of views), suggesting a high level of interest in gathering information about evidence-based and promising practices for K–12 PEI programs.
Some networks and collaborations among agencies and organizations supporting SMH are highly valuable and relatively cost-effective to sustain. The RAND team found that some collaborative groups have been able to influence policy or create and disseminate products with widespread impact across institutions and communities. For instance, an SMH Policy Workgroup convened by the California Department of Education (CDE) capitalized on a time-sensitive opportunity to provide feedback on the State teacher credentialing standards. The Workgroup put forth an official recommendation for new standards that require educators to be trained to provide students and their families with greater access to mental health services, and to enhance collaborative partnerships to link students to appropriate services.
Our Campus-Wide Survey results indicate that for students who sought or were referred to services, a much smaller proportion of CCC students accessed services compared to UC and California State University (CSU) systems (41 percent versus 71–77 percent). CCC campus policy regarding service funding and availability is governed by the local Board of Trustees, thus the availability of mental health services is inconsistent across campuses. When mental health services are not available on campuses, campus collaboration with community agencies and county mental health can be vital to increase access and usage.
Upcoming evaluation work. The evaluation team is currently analyzing results from a networking and collaboration survey administered to campus grant coordinators, advisory committee members, and K–12 policy workgroup and county consortia members. These results may reveal participants' perspectives regarding which collaborative structures and governance formats are most effective for increasing SMH service access, quality, coordination, and sustainability—as well as what characteristics of the collaborative groups (e.g., formality, definition of roles, accountability) were associated with outcomes. These results, which will be available by fall of 2014, may be helpful in fine-tuning recommendations for which kinds of networks and collaborations to sustain.
Higher education systems seem interested in and ready to use information from the higher education climate survey (administered to students and staff/faculty on campuses) to help inform future higher education activities and to monitor the status of student needs, which argues for sustained assessment. The UC online Interactive Screening Program (ISP) is also a sustainable tool to identify student mental health needs. The ISP screening plus the higher education surveys would be a powerful combination of assessment and accountability that would be relatively inexpensive to sustain.
Beyond the three initiative areas, we present our recommendations for sustained evaluation, which are applicable across initiatives. These include program-level evaluation, population surveillance, and targeted effectiveness studies.
As current evaluation efforts are completed, programs should transition to maintaining internal, core evaluation capacities for both external reporting purposes (e.g., reporting reach and short-term outcomes of ongoing PEI activities to CalMHSA) and to support continuous quality improvement efforts. Programs will vary in their need for resources and technical assistance to further develop these capacities. Investments to date in RAND and program partner evaluations have provided a strong basis for approaches and tools that can be utilized to support sustained, routine program-level evaluation.
Ongoing and long-term population surveillance is needed to assess whether a complex, multi-level, and interactive set of PEI strategies is reaching those at higher risk of mental health problems and achieving the longer-term goals of preventing suicide, reducing stigma and discrimination, and improving student mental health. We recommend efforts to coordinate with and improve existing population surveillance systems so that they better support population tracking related to mental health PEI activities. Such efforts might include the following actions:
- Improve standards in California for investigating and reporting death by suicide (adopting national recommendations)
- Enhance the California Health Interview Survey (CHIS), for example, to improve or add measures of stigma, willingness to intervene and assist with mental health problems, recovery beliefs, attitudes about and willingness to use hotlines, and to update the K6 distress measure (we made these suggestions to the CHIS in 2012)
- Enhance the K–12 California School Climate, Health, and Learning Survey (CAL-SCHLS) by incorporating key items assessing the student mental health climate and activities into the core module or other commonly used modules
- Sustain higher education campus-wide surveys to monitor the ongoing activities and climate related to student mental health issues. Our previous higher education survey identified substantial impairment due to mental health concerns in the college student population (Sontag-Padilla et al., 2014), underlining the importance of continued monitoring.
Targeted Effectiveness Studies
The evidence base for PEI programs is weak or nonexistent for some approaches that may be particularly promising or important for California's diverse populations (see Acosta et al., 2012; Collins et al., 2012; Stein et al., 2012). We suggest considering investment in strategically focused and rigorously conducted effectiveness studies when this information would be highly useful to informing broad dissemination decisions. For example, program partner NAMI is currently restructuring a key training program to culturally and linguistically adapt it for use with California's diverse population. This ambitious effort is highly innovative, but like all promising new efforts, it requires evaluation to establish its effectiveness.
Overall, we see significant value in sustaining many mental health PEI programs already funded by CalMHSA, and note the empirical evidence for several programs based on the research literature and RAND evaluation efforts to date. We also note some instances where increased funding in new areas could fill a gap in CalMHSA programming (e.g., restricting access to means to die by suicide). These preliminary recommendations are offered at an interim stage of the evaluation to assist in decisionmaking regarding how to invest CalMHSA resources. Additional findings from the evaluation will be presented when they become available.
Acosta, Joie D., Rajeev Ramchand, Lisa H. Jaycox, Amariah Becker, and Nicole K. Eberhart, IInterventions to Prevent Suicide: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1317-CMHSA, 2012. As of July 29, 2014:
Acosta, Joie D., and Rajeev Ramchand, Adults Exposed to “Know the Signs” Are More Confident Intervening with Those At Risk for Suicide, Santa Monica, Calif.: RAND Corporation, RR-686-CMHSA, 2014. As of July 29, 2014:
Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, “What Has the CalMHSA Statewide Mental Health Prevention and Early Intervention Program Done So Far? Key Results from the Baseline RAND General Population Survey,” Santa Monica, Calif.: RAND Corporation, RB-9771-CMHSA, 2014. As of July 29, 2014:
Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, Dana Schultz, and Nicole K. Eberhart, Interventions to Reduce Mental Health Stigma and Discrimination: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1318-CMHSA, 2012. As of July 29, 2014:
Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, and Elizabeth Roth, Racial and Ethnic Differences in Mental Illness Stigma in California, Santa Monica, Calif.: RAND Corporation, RR-684-CMHSA, 2014. As of July 29, 2014:
Ramchand, Rajeev, and Elizabeth Roth, Racial and Ethnic Differences in Exposure to Suicide Prevention Messaging, Confidence in One's Ability to Intervene with Someone at Risk, and Resource Preferences, Santa Monica, Calif.: RAND Corporation, RR-682-CMHSA, 2014. As of July 29, 2014:
Sontag-Padilla, Lisa, Elizabeth Roth, Michelle W. Woodbridge, Courtney Ann Kase, Karen Chan Osilla, Elizabeth D'Amico, Lisa H. Jaycox, and Bradley D. Stein, CalMHSA Student Mental Health Campus-Wide Survey: 2013 Summary Report, Santa Monica, Calif.: RAND Corporation, RR-685-CMHSA, 2014. As of July 29, 2014:
Stein, Bradley D., Lisa M. Sontag-Padilla, Karen Chan Osilla, Michelle W. Woodbridge, Courtney Ann Kase, Lisa H. Jaycox, Elizabeth D'Amico, Jennifer L. Cerully, Nicole K. Eberhart, and Shari Golan, Interventions to Improve Student Mental Health: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1319-CMHSA, 2012. As of July 29, 2014:
This research was conducted in RAND Health, a division of the RAND Corporation.