Evaluation of California's Statewide Mental Health Prevention and Early Intervention Programs

Summary of Key Year 2 Findings

by Nicole K. Eberhart, M. Audrey Burnam, Sandra H. Berry, Rebecca L. Collins, Patricia A. Ebener, Rajeev Ramchand, Bradley D. Stein, Michelle W. Woodbridge

This Article

RAND Health Quarterly, 2015; 5(1):15

Abstract

This article summarizes key findings from the second year of an evaluation of the California Mental Health Services Authority’s statewide prevention and early intervention programs.

For more information, see RAND RR-971-CMHSA at https://www.rand.org/pubs/research_reports/RR971.html

Full Text

Mental, emotional, and behavioral disorders are highly prevalent, affecting approximately one in four young people and one in five adults over the course of a year. These disorders result in high economic costs to society and exact a heavy personal and family toll on those who experience them. For over a decade, public health leaders and mental health experts have called for development and dissemination of prevention and early intervention (PEI) approaches that promote mental health, reduce the risk of mental disorders, and diminish the severity and negative consequences associated with onset of mental, emotional, and behavioral disorders (Mrazek and Haggerty, 1994; O'Connell, Boat, and Warner, 2009; U.S. Department of Health and Human Services, 1999). A recent article from the Substance Abuse and Mental Health Services Administration (SAMHSA) emphasized that a comprehensive public health approach is required to promote mental health and prevent illness, and that intervention targets should include strengthening individuals and families by building their resilience, as well as improving the health of the communities in which they live (Power, 2010).

In California, the passage of Proposition 63 set aside funds for PEI and allowed for local and statewide activities. Recognizing the importance of including evidence-based PEI approaches into a comprehensive spectrum of care for mental disorders, the California Mental Health Services Authority (CalMHSA)—a coalition of California counties designed to provide economic and administrative support to mental health service delivery—formed a statewide PEI Implementation Workplan. The workplan guided the development of three program initiatives designed to meet the following goals: (1) stigma and discrimination reduction, (2) suicide prevention, and (3) improvement of student mental health. Under each initiative, community organizations, private firms, and public K–12 and higher education systems serve as PEI program partners, performing activities to meet the initiative's goals. In the summer of 2011, program partners began their activities.

In the fall of 2011, the RAND Corporation was selected to design and implement an evaluation of the three initiatives, and the final evaluation plan was approved and launched in the summer of 2012. RAND has previously published summaries of its year one evaluation findings for the summer of 2012 through the summer of 2013 (e.g., Burnam et al., 2014b; Collins et al., 2014a; Ramchand et al., 2014; Stein et al., 2014). What follows focuses on the evaluation findings that have been produced in the second year of the evaluation, from the summer of 2013 through the summer/fall of 2014. Naturally, the timing of the evaluation is governed, in part, by the pace of the program partner activities being evaluated.

Key Findings: Positive Outcomes

Based on results in year two, the CalMHSA PEI initiatives are successfully launched and are already showing positive outcomes in stigma and discrimination reduction, suicide prevention, and improvement of student mental health.

Stigma and Discrimination Reduction

Social Marketing Campaign
  • The Each Mind Matters social marketing campaign and the associated website show promise in reaching California adults.
  • After attending the “Walk in Our Shoes” theatrical presentation, middle-school students expressed less stigmatizing attitudes in a variety of domains.

Suicide Prevention

Social Marketing Campaign
  • The Know the Signs suicide prevention campaign was strongly aligned with best practices and is one of the best media campaigns on the subject.
  • Those who reported exposure to Know the Signs campaign materials report being more confident in intervening with those at risk of suicide.
Training in Suicide Intervention Skills Delivered In-Person to Target Audiences
  • Fidelity to training topics in the Applied Suicide Intervention Skills Training (ASIST) was high.

Student Mental Health

Training School Faculty/Staff/Students
  • PEI trainings reached large numbers of individuals, particularly women and individuals from diverse racial/ethnic backgrounds.
  • We found strong evidence of benefits from trainings within both the higher education and K–12 systems.
Online Resources
  • The targeted campaign to K–12 and higher education stakeholders in California was successful at engaging Californians, as intended.
Collaboration/Networking
  • At least half of the respondents in all groups reported that they were planning for sustainability and believed their collaborations would remain strong even after the CalMHSA support ended.
  • More than 60 percent of respondents attributed “a great extent” of their capacity to CalMHSA's support.

Key Findings: Informing Future Program Planning

  • Asian Americans reported relatively higher levels of stigmatizing attitudes toward individuals with mental illness and exposure to Know the Signs suicide prevention messages was significantly lower among Asian Americans than other racial/ethnic groups.
  • Young adults hold some of the least-stigmatizing attitudes toward mental illness, are more likely to know someone with mental illness and are less likely to feel they know how to help.
  • One in five higher education students reported probable psychological distress, and high numbers of students reported impairment in academic performance associated with anxiety or depression. However, less than 40 percent of higher education faculty/staff believed they have the skills to directly help students with mental health problems.
  • While fidelity to training topics was high in ASIST training, there is room for improvement in the style of presentation.

General Conclusions

Based on results in year two, the CalMHSA PEI initiatives are successfully launched and are already showing positive outcomes in stigma and discrimination reduction, suicide prevention, and improvement of student mental health. The positive results at this early stage are particularly encouraging since many key effects of PEI programming cannot be detected immediately.

A public health approach to prevention—engaging individuals, families, and communities to avoid illness and promote mental health—requires a comprehensive and sustained effort. Overall, the RAND evaluation has documented that the CalMHSA initiatives have been successfully launched, showing significant progress in program implementation. Accumulating evidence for program effectiveness in shifting intended short-term outcomes (such as knowledge and attitudes) related to stigma and discrimination reduction, suicide prevention, and improvement of student mental health are promising. The evaluation has found evidence that CalMHSA's statewide initiatives are reaching targeted California populations, reducing mental illness stigma, increasing the number of Californians with the skills to intervene with and refer individuals with mental health challenges, and disseminating evidence-based practices through online resources and strategic collaborations. CalMHSA investments in monitoring mental health and mental health stigma and treatment needs in the broader population of California adults and in assessing the mental health and campus climate of college students provide baseline measures for assessing longer-term impacts of PEI efforts in California.

In this document, we provide summaries of evaluation findings from the second year of evaluation for each of the three initiatives, and also summarize investments in and results from statewide evaluation efforts that span the three initiatives. Future evaluation will include further assessment of program implementation and reach, short-term outcomes, and results from statewide population monitoring of PEI impacts.

Stigma and Discrimination Reduction Initiative

Mental illness stigma refers to negative stereotypes, prejudice, and discrimination toward those with mental health challenges. Based on population surveys about public knowledge of and response to mental illness, we know that stigma is common in the United States (Pescosolido et al., 2010). Stigma and discrimination reduction is a key component of PEI approaches to mental disorders because stigma has been shown to increase negative emotional states for those who experience mental health issues, as well as discourage these individuals from seeking and receiving treatment, diminish social support and increase risk of social isolation, and result in challenging interactions with or discrimination from those with influence (such as employers, landlords, health care providers, and law enforcement). Approaches to stigma reduction include broad, multi-component mass media campaigns that focus on changing public attitudes, and more-focused trainings to educate and change attitudes and behavior in specific target groups, such as students, police officers, and health providers (Collins et al., 2012).

In the CalMHSA Stigma and Discrimination Reduction Initiative, PEI activities include a social marketing campaign, trainings, and online resources. We discuss recent evaluation results for the social marketing campaign below. Training and resource evaluations are ongoing and will be reported on later this year.

Social Marketing Campaign

Each Mind Matters is a social marketing campaign launched in 2012 that seeks to reduce the stigma associated with mental illness. The campaign was developed based on the research literature on best practices in stigma reduction, and it includes four sub-parts, each with a different targeted age group, subgoals, and strategies designed to fit that age. The campaign was branded “Each Mind Matters” in 2013 and has since been extended to cover a variety of CalMHSA's stigma reduction activities. RAND's first-year evaluation findings were that both the Each Mind Matters social marketing campaign and the associated website show promise in reaching California adults. Specifically, a RAND survey of California adults found a modest level of exposure to Each Mind Matters within the first month of the campaign; 11 percent of those surveyed said they had seen or heard the phrase “Each Mind Matters,” the new (at that time) branding of CalMHSA efforts. Although less than 1 percent reported visiting the Each Mind Matters website, at the time of the survey it was just beginning to serve as the hub for dissemination of CalMHSA resources and was primarily the host of the “A New State of Mind” documentary (Burnam et al., 2014a). In year two of the evaluation, more-detailed analysis of the statewide baseline survey, which oversampled diverse groups, indicated that Asian Americans reported relatively high levels of stigmatizing attitudes toward individuals with mental illness (see Figure 1), as did Latinos who were interviewed in Spanish (rather than English), compared with other groups (Collins et al., 2014c). This suggests that stigma reduction efforts should continue to reach out to these groups.

Figure 1. Asian Americans Are More Unwilling to Have Contact with Someone Experiencing Mental Illness, in Some Contexts

Figure 1. Asian Americans Are More Unwilling to Have Contact with Someone Experiencing Mental Illness, in Some Contexts

Conversely, additional analysis of the statewide survey results indicated that young adults hold some of the least-stigmatizing attitudes toward mental illness (Collins et al., 2014b). They are also the most likely to report contact with someone with mental illness in the past year. At the same time, however, fewer of them say they linked those individuals to resources like treatment. These results suggest that the Each Mind Matters campaign's messaging targeting young adults should focus on how they can provide support to individuals with mental health concerns and link them to resources, with relatively less need for stigma reduction messaging.

The social marketing campaign also includes programs related to stigma reduction. The “Walk in Our Shoes” program includes a theatrical presentation for students, an informational website with educational materials, and teacher curriculum for follow-up to the program. RAND's evaluation, which focused on the theatrical presentation, found that middle-school students who attended the “Walk in Our Shoes” presentation expressed less stigmatizing attitudes in a variety of domains afterward (Wong et al., 2014). After viewing this brief theatrical presentation, students expressed greater willingness to interact with fellow students with a mental health problem and more positive emotional responses to a hypothetical student with a mental health problem. The presentation was evaluated with predominately Latino participants, and 81 percent said they would recommend the presentation to someone of their cultural background.

The RAND team is currently analyzing follow-up survey data that will provide estimates of the degree to which exposure to the social marketing campaign has changed over time, and the extent to which Each Mind Matters is improving reach to diverse ethnic and racial groups. Estimates of campaign effectiveness in reducing stigma toward mental illness in California will also be part of that report. Analyses of the California Well-Being Survey will examine reach of the social marketing campaign to people experiencing mental health challenges in the state and provide benchmarking estimates of self-stigma and experienced discrimination in this key group. RAND's upcoming message-testing experiments will evaluate which social marketing messages from Each Mind Matters are most effective in stigma reduction.

Suicide Prevention Initiative

In California, over 3,000 people die by suicide each year. The three strongest risk factors for suicide are prior suicide attempts, mental disorders, and substance use/abuse. Approaches to suicide prevention include reducing access to lethal means, responding to crises effectively, providing high-quality mental health care to those with mental health needs, and increasing awareness, knowledge, and skills among those who may have opportunities to intervene and facilitate access to appropriate care (Acosta et al., 2012).

In the CalMHSA Suicide Prevention Initiative, programs fell into three major categories: a social marketing campaign, training, and crisis lines. We discuss recent evaluation results for each of these types of PEI activities.

Social Marketing Campaign

In 2011, CalMHSA implemented a statewide mass media effort called Know the Signs that aims to prepare more Californians to prevent suicide by encouraging them to know the warning signs, offer support to persons at risk, and reach out to local resources. The program's slogan is “Pain Isn't Always Obvious.” Messages are promoted through common media channels (posters, banner ads, billboards) that encourage people to visit the campaign website (www.suicideispreventable.org), where potential helpers can learn about the warning signs for suicide and available resources. RAND's evaluation of the Know the Signs suicide prevention social marketing campaign is three-pronged: An expert panel evaluated the quality of the campaign; a statewide survey evaluated exposure to the campaign and whether those exposed have increased knowledge, help-seeking, and awareness; and a message-testing experiment will test the effectiveness of Know the Signs messages. This year's evaluation documented some promising findings for Know the Signs in the first two domains, with results for the third forthcoming.

An expert panel convened by RAND determined that the Know the Signs suicide prevention campaign was strongly aligned with best practices and is one of the best media campaigns on the subject, although there are still some areas that could be improved (Acosta and Ramchand, 2014b). RAND's statewide survey had previously found a relatively high level of exposure to Know the Signs campaign materials. More-detailed analyses of the baseline survey indicated that exposure to Know the Signs messages was significantly lower among Asian Americans than other racial/ethnic groups (Ramchand and Roth, 2014b), and Latinos and Asians who took the survey in a language other than English were less likely to report being exposed to the Know the Signs campaign compared to those who took the survey in English (Ramchand and Roth, 2014a). Importantly, those who reported exposure to Know the Signs campaign materials report being more confident in intervening with those at risk of suicide (see Figure 2; Acosta and Ramchand, 2014a). They felt more comfortable discussing suicide, reported greater awareness of the warning signs, and reported greater skills and knowledge relating to intervening with or referring someone at risk.

Figure 2. Adults Exposed to Know the Signs Are More Confident Intervening with Those at Risk for Suicide

Figure 2. Adults Exposed to Know the Signs Are More Confident Intervening with Those at Risk for Suicide

RAND is conducting a follow-up statewide survey that will enable further tracking of exposure to the Know the Signs campaign and its effects. Message-testing experiments will also be conducted that can inform the development of effective messages going forward. In addition, RAND will evaluate how the media covered suicides in the period before and after the Know the Signs social marketing campaign, which included a component that trained media writers on responsible reporting of suicide. This analysis will compare California media to nationally circulated publications.

Training in Suicide Intervention Skills Delivered In-Person to Target Audiences

Applied Suicide Intervention Skills Training (ASIST) is an evidence-based training that provides individuals with knowledge and skills to intervene with individuals who may be at risk of suicide. As part of the California Suicide Prevention Initiative, there was a significant investment in increasing California's capacity to provide ASIST workshops by training individuals to become ASIST trainers. RAND developed a fidelity and adherence protocol to monitor the quality and delivery of ASIST workshops given by newly trained trainers. The RAND team observed and rated five multi-day workshops and found that fidelity to the ASIST training topics was high, meaning that trainers tended to cover all aspects of the manual-based training. However, adherence to ASIST competencies was not as high (i.e., trainers did not consistently follow the recommended style of presenting), suggesting that trainers may need to improve their efforts to tailor content to specific audiences, promote cultural competence, and cover topics within the recommended time allotments (Osilla et al., 2014).

RAND is currently analyzing the results of post-training surveys to examine the ways in which trainees benefit from the trainings.

Crisis Lines

National and local crisis lines (telephone hotlines) exist throughout the United States, providing a number that an individual in distress can call to get immediate support and help in accessing care. As part of the Suicide Prevention Initiative, some California crisis centers received support to improve or expand their existing hotlines, and one new hotline was developed. RAND's evaluation of crisis lines includes a statewide survey to examine whether the lines are reaching those in need and an examination of the extent to which crisis lines are delivering high-quality services. Our statewide survey examined where people would like to get help if they were suicidal. We found that people would generally prefer face-to-face help from either a mental health professional or family/friends, but the majority (62 percent) would still be likely to seek help from a crisis hotline (Becker and Ramchand, 2014). However, the preference for in-person resources was not as strong among Latinos and African Americans, who expressed more interest in text and web-based chat services as compared to Whites (Ramchand and Roth, 2014b).

To assess crisis call quality, researchers observed live calls and rated multiple aspects of them using a call-monitoring protocol that was specifically developed and tested for this purpose. RAND conducted live call monitoring at ten crisis centers and analyses are under way. The call-monitoring protocol will also be made available online so that it may be used by any hotline to support quality improvement.

Finally, RAND is developing a policy brief that presents the opportunities and challenges that are associated with alternative systems for providing telephone-delivered acute crisis care.

Student Mental Health Initiative

Student mental health PEI approaches are generally designed to promote a school and campus climate reflecting high awareness and sensitivity to mental health issues among students of all ages—for example, by supporting students coping with emotional/behavioral issues and stress and encouraging students to seek help. Some programs focus on providing faculty, staff, peer leaders, or family members with knowledge and skills to support, identify, and respond to specific mental health issues or populations (Stein et al., 2012).

In the CalMHSA Student Mental Health Initiative, programs fell into three major categories: trainings, online resources, and networking/collaboration activities. The K–12 program partners facilitating these activities include the California Department of Education (CDE) and the California County Superintendents Educational Services Association (CCSESA). The higher education program partners include the California Community Colleges (CCC), California State University (CSU), and University of California (UC) systems. We summarize recent evaluation results for each of the major PEI activity categories below.

Training School Faculty/Staff/Students

A large number of mental health PEI trainings were administered to staff in California's K–12 schools and staff and students in California's higher education system. Some of these trainings were informed by research literature on best practices. Examples include TETRIS (Training Educators Through Recognition and Identification Strategies) training, which focuses on increasing knowledge of student risk and protective factors, school and community resources, intervention strategies, and ways to promote mentally healthy learning environments, which was implemented by CDE, and standardized, manualized trainings, including Mental Health First Aid (MHFA), Question Persuade Refer (QPR), and ASIST, which were implemented by UC and CSU.

RAND evaluated a subset of trainings hosted by each of the student mental health program partners. PEI trainings were evaluated using (1) surveys administered at the time of the training to K–12 and higher education staff and faculty and college/university students and (2) RAND's Higher Education Campus-Wide Survey, which was administered to large samples of higher education staff, faculty, and students across campuses. RAND used surveys administered at the trainings to examine both reach/penetration and short-term outcomes (i.e., potential benefits) of a subset of the many PEI trainings being conducted by higher education and K–12 program partners through the CalMHSA statewide initiatives. In both systems, we found that the PEI trainings reached large numbers of individuals, particularly women and individuals from diverse racial/ethnic backgrounds (Osilla et al., 2015a; Osilla et al., 2015b). In the higher education trainings, the majority of participants were students, whereas most participants in the K–12 trainings were teachers. We found strong evidence of benefits from trainings within both the higher education and K–12 systems, with participants reporting greater confidence to intervene with students in distress; greater confidence to refer students to mental health resources; and greater self-reported likelihood to intervene or refer students in distress (see Figure 3). Participants felt the trainings were good quality, helpful, and important to attend.

Figure 3. Training Increased Reported Confidence/Likelihood of Intervening and Referring

Figure 3. Training Increased Reported Confidence/Likelihood of Intervening and Referring

However, there is evidence that not everyone who could benefit from trainings is receiving them. RAND's Higher Education Campus-Wide Survey (Sontag-Padilla et al., 2014) found that almost one in five students reported probable psychological distress, and high numbers of students reported impairment in academic performance associated with anxiety or depression (see Figure 4). These rates of psychological distress are consistent with other studies of higher education populations (Hunt and Eisenberg, 2010). Rates of impairment associated with mental health problems are higher than those reported elsewhere, but this could be due to differences in how impairment was assessed (Keyes et al., 2012). Nonetheless, it is notable that less than 40 percent of faculty/staff believed they have the skills to directly help students with mental health problems. Yet, on average, only a subset of staff/faculty (12–29 percent across the higher education systems) participated in student mental health–related training in the six months prior to the survey. The largest barrier to participation was lack of knowledge that the trainings were offered. These results suggest the importance of continuing student mental health training efforts, as there is evidence for both short-term effectiveness of trainings and need for trainings, and it takes time to see long-term impacts on such large educational systems.

Figure 4. Substantial Proportions of Students Indicate Impaired Academic Performance Due to Mental Health Issues

Figure 4. Substantial Proportions of Students Indicate Impaired Academic Performance Due to Mental Health Issues

RAND will continue to evaluate reach and short-term outcomes for key student mental health trainings. In addition, forthcoming results of a survey focused on California K–12 principals' perceptions of pressing student problems and school PEI activities related to student mental health will also inform the student mental health trainings evaluation.

Online Resources

RAND's evaluation of the K–12 and higher education websites demonstrated that program partners' websites have experienced substantial use, with large numbers of visits and page views (see Burnam et al., 2014c). Over 90 percent of visits to the systems' websites originated from within California, suggesting that the targeted campaign to K–12 and higher education stakeholders in California was successful at engaging Californians, as intended. Additionally, the use of search engines to access the website increased across the reporting periods, suggesting an increase in awareness of the website and use of search terms related to site content. Promotional campaigns have also successfully increased website traffic.

RAND will continue to examine usage of key online resources, including how their use changes over time.

Collaboration/Networking

The RAND team, led by collaborators at SRI International, found that some collaborations focused on improving student mental health have been able to influence policy or create and disseminate products with widespread impact across institutions and communities (Woodbridge et al., 2014a; Woodbridge et al., 2014b). Recent results from a networking and collaboration survey administered to CalMHSA campus grant coordinators, student mental health advisory committee members, and the statewide K–12 policy workgroup and county consortia members revealed some of the facilitators and barriers to successful collaborations (Woodbridge et. al., 2015a; Woodbridge et al., 2015b). Overwhelmingly, respondents in all the groups surveyed reported that the advantages of collaboration outweighed the disadvantages. Further, many respondents attributed improvements in SMH outcomes to collaboration. Finally, at least half of the respondents in all groups reported that they were planning for sustainability and believed their collaborations would remain strong even after the CalMHSA support ended. When asked the degree to which CalMHSA provided resources necessary to pursue collaborative activities that otherwise would not have occurred, more than 60 percent of respondents attributed “a great extent” of their capacity to CalMHSA's support.

Evaluation of Long-Term Effects

Often the most meaningful effects of PEI programming cannot be detected immediately; it takes time for impacts on the targeted outcomes to become apparent. As such, ongoing and long-term population surveillance is needed to assess whether a complex, multi-level, and interactive set of PEI strategies is reaching those at higher risk of mental health problems and achieving the longer-term goals of preventing suicide, reducing stigma and discrimination, and improving student mental health (see Freeman et al., 2010).

Thus, in addition to evaluating the short-term effects of specific programs, RAND has also implemented a number of statewide surveys targeting different populations (all Californians, those who are at higher risk due to recent symptoms of mental illness, and students/staff/faculty) to lay the groundwork for assessing key, longer-term effects of the three PEI initiatives, and to inform program planning with respect to population needs. Some of the key results from these surveys are outlined above. As a result of CalMHSA's investments in tracking the mental health of Californians, RAND's evaluation results can be used as a tool for tracking the longer-term impacts of investments in PEI, and for planning at the statewide level. For instance, campus-wide surveys are being used to monitor ongoing activities and climate related to student mental health issues among higher education students. RAND's baseline survey identified substantial impairment due to mental health concerns in the college student population (Sontag-Padilla et al., 2014), underlining the importance of continuing to monitor mental health–related impairment in this population.

In the future, RAND will present findings from

  • a one-year follow-up survey of California adults
  • a one-year follow-up survey of California higher education students, staff, and faculty
  • a survey of California K–12 principals focused on understanding their perceptions of pressing student problems and school PEI activities related to student mental health
  • a targeted survey focusing on California adults with recent symptoms of mental illness
  • an analysis of mental health and stigma data collected through the California Health Interview Survey (CHIS).

Results from these surveys will be used to inform recommendations for ongoing surveillance of the mental health of Californians.

References

Acosta, Joie D., and Rajeev Ramchand, Adults Exposed to “Know the Signs” Are More Confident Intervening with Those at Risk for Suicide, Santa Monica, Calif.: RAND Corporation, RR-686-CMHSA, 2014a. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR686.html

Acosta, Joie D., and Rajeev Ramchand, “Know the Signs” Suicide Prevention Media Campaign Is Aligned with Best Practices and Highly Regarded by Experts, Santa Monica, Calif.: RAND Corporation, RR-818-CMHSA, 2014b. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR818.html

Acosta, Joie D., Rajeev Ramchand, Lisa H. Jaycox, Amariah Becker, and Nicole K. Eberhart, Interventions to Prevent Suicide: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1317-CMHSA, 2012. As of January 26, 2015:
http://www.rand.org/pubs/technical_reports/TR1317.html

Becker, Amariah, and Rajeev Ramchand, Where Would California Adults Prefer to Get Help If They Were Feeling Suicidal? Santa Monica, Calif.: RAND Corporation, RR-681-CMHSA, 2014. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR681.html

Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, “What Has the CalMHSA Statewide Mental Health Prevention and Early Intervention Program Done So Far? Key Results from the Baseline RAND General Population Survey,” Santa Monica, Calif.: RAND Corporation, RB-9771-CMHSA, 2014a. As of January 26, 2015:
http://www.rand.org/pubs/research_briefs/RB9771.html

Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, “What Has the CalMHSA Statewide Mental Health Prevention and Early Intervention Program Done So Far? Summary and Commentary for Year 1 Evaluation,” Santa Monica, Calif.: RAND Corporation, RB-9772-CMHSA, 2014b. As of January 26, 2015:
http://www.rand.org/pubs/research_briefs/RB9772.html

Burnam, M. Audrey, Sandra H. Berry, Nicole K. Eberhart, Rebecca L. Collins, Patricia A. Ebener, Rajeev Ramchand, Bradley D. Stein, and Michelle W. Woodbridge, Recommendations for Sustaining California's Statewide Mental Health Prevention and Early Intervention Programs, Santa Monica, Calif.: RAND Corporation, RR-787-CMHSA, 2014c. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR787.html

Collins, Rebecca L., Jennifer L. Cerully, Eunice C. Wong, Shari Golan, Jennifer Yu, and Gabrielle Filip-Crawford, “What Has the Stigma and Discrimination Reduction Initiative Done So Far? Year 1 Findings,” Santa Monica, Calif.: RAND Corporation, RB-9756-CMHSA, 2014a. As of January 26, 2015:
http://www.rand.org/pubs/research_briefs/RB9756.html

Collins, Rebecca L., Elizabeth Roth, Jennifer L. Cerully, and Eunice C. Wong, Beliefs Related to Mental Illness Stigma Among California Young Adults, Santa Monica, Calif.: RAND Corporation, RR-819-CMHSA, 2014b. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR819.html

Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, and Elizabeth Roth, Racial and Ethnic Differences in Mental Illness Stigma in California, Santa Monica, Calif.: RAND Corporation, RR-684-CMHSA, 2014c. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR684.html

Collins, Rebecca L., Eunice C. Wong, Jennifer L. Cerully, Dana Schultz, and Nicole K. Eberhart, Interventions to Reduce Mental Health Stigma and Discrimination: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1318-CMHSA, 2012. As of January 26, 2015:
http://www.rand.org/pubs/technical_reports/TR1318.html

Freeman, E. J., L. J. Colpe, T. W. Strine, S. Dhingra, L. C. McGuire, L. D. Elam-Evans, and G. S. Perry, “Public Health Surveillance for Mental Health,” Preventing Chronic Disease, Vol. 7, No. 1, 2010, p. A17.

Hunt, J., and D. Eisenberg, “Mental Health Problems and Help-Seeking Behavior Among College Students,” Journal of Adolescent Health, Vol. 46, No. 1, 2010, pp. 3–10.

Keyes, C. L., D. Eisenberg, G. S. Perry, S. R. Dube, K. Kroenke, and S. S. Dhingra, “The Relationship of Level of Positive Mental Health with Current Mental Disorders in Predicting Suicidal Behavior and Academic Impairment in College Students,” Journal of American College Health, Vol. 60, No. 2, 2012, pp. 126–133.

Mrazek, Patricia J., and Robert J. Haggerty, eds., Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research, Washington, D.C.: The National Academies Press, 1994.

O'Connell, Mary Ellen, Thomas Boat, and Kenneth E. Warner, eds., Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities, Washington, D.C.: The National Academies Press, 2009.

Osilla, Karen Chan, Dionne Barnes-Proby, Mary Lou Gilbert, and Rajeev Ramchand, A Case Study Evaluating the Fidelity of Suicide Prevention Workshops in California, Santa Monica, Calif.: RAND Corporation, RR-755-CMHSA, 2014. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR755.html

Osilla, Karen Chan, Asha Goldweber, Rachana Seelam, Courtney Ann Kase, Elizabeth Roth, and Bradley D. Stein, Mental Health Trainings for School Staff in California's K–12 System Are Associated with Increased Confidence and Likelihood to Intervene with and Refer Students, Santa Monica, Calif.: RAND Corporation, RR-953-CMHSA, 2015a. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR953.html

Osilla, Karen Chan, Michelle Woodbridge, Rachana Seelam, Courtney Ann Kase, Elizabeth Roth, and Bradley D. Stein, Mental Health Trainings in California's Higher Education System Are Associated with Increased Confidence and Likelihood to Intervene with and Refer with Students, Santa Monica, Calif.: RAND Corporation, RR-954-CMHSA, 2015b. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR954.html

Pescosolido, B. A., J. K. Martin, J. S. Long, T. R. Medina, J. C. Phelan, and B. G. Link, “A Disease Like Any Other? A Decade of Change in Public Reactions to Schizophrenia, Depression, and Alcohol Dependence,” American Journal of Psychiatry, Vol. 167, No. 11, 2010, pp. 1321–1330.

Power, A. Kathryn, “Transforming the Nation's Health: Next Steps in Mental Health Promotion,” American Journal of Public Health, Vol. 100, 2010, pp. 2343–2346.

Ramchand, Rajeev, Joie D. Acosta, Amariah Becker, Patricia A. Ebener, Lisa H. Jaycox, and Karen Chan Osilla, “What Has the Suicide Prevention Initiative Done So Far? Year 1 Findings,” Santa Monica, Calif.: RAND Corporation, RB-9727-CMHSA, 2014. As of January 26, 2015:
http://www.rand.org/pubs/research_briefs/RB9757.html

Ramchand, Rajeev, and Elizabeth Roth, Language Differences in California Adults' Exposure to Suicide Prevention Messaging, Confidence in One's Ability to Intervene with Someone at Risk, and Resource Preferences, Santa Monica, Calif.: RAND Corporation, RR-754-CMHSA, 2014a. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR754.html

Ramchand, Rajeev, and Elizabeth Roth, Racial and Ethnic Differences in Exposure to Suicide Prevention Messaging, Confidence in One's Ability to Intervene with Someone at Risk, and Resource Preferences, Santa Monica, Calif.: RAND Corporation, RR-682-CMHSA, 2014b. As of July 29, 2014:
http://www.rand.org/pubs/research_reports/RR682.html

Sontag-Padilla, Lisa, Elizabeth Roth, Michelle W. Woodbridge, Courtney Ann Kase, Karen Chan Osilla, Elizabeth D'Amico, Lisa H. Jaycox, and Bradley D. Stein, CalMHSA Student Mental Health Campus-Wide Survey: 2013 Summary Report, Santa Monica, Calif.: RAND Corporation, RR-685-CMHSA, 2014. As of July 29, 2014:
http://www.rand.org/pubs/research_reports/RR685.html

Stein, Bradley D., Lisa Sontag-Padilla, Karen Chan Osilla, Michelle W. Woodbridge, Courtney Ann Kase, Lisa H. Jaycox, Elizabeth D'Amico, Jennifer L. Cerully, Nicole K. Eberhart, and Shari Golan, Interventions to Improve Student Mental Health: A Literature Review to Guide Evaluation of California's Mental Health Prevention and Early Intervention Initiative, Santa Monica, Calif.: RAND Corporation, TR-1319-CMHSA, 2012. As of January 26, 2015:
http://www.rand.org/pubs/technical_reports/TR1319.html

Stein, Bradley D., Michelle W. Woodbridge, Lisa Sontag-Padilla, Karen Chan Osilla, Courtney Ann Kase, Asha Goldweber, Lisa H. Jaycox, and Elizabeth D'Amico, “What Has the Student Mental Health Initiative Done So Far? Year 1 Findings,” Santa Monica, Calif.: RAND Corporation, RB-9757-CMHSA, 2014. As of January 26, 2015:
http://www.rand.org/pubs/research_briefs/RB9757.html

U.S. Department of Health and Human Services, Mental Health: A Report of the Surgeon General, Rockville, Md., 1999.

Wong, Eunice C., Jennifer L. Cerully, Rebecca L. Collins, and Elizabeth Roth, Findings from the School-Based Theatrical Performance “Walk In Our Shoes,” Santa Monica, Calif.: RAND Corporation, RR-683-CMHSA, 2014. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR683.html

Woodbridge, Michelle W., Asha Goldweber, Jennifer Yu, Shari Golan, and Bradley D. Stein, California K–12 Schools and Communities Collaborate to Support Student Mental Health, Santa Monica, Calif.: RAND Corporation, RR-688-CMHSA, 2014a. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR688.html

Woodbridge, Michelle W., Asha Goldweber, Jennifer Yu, Shari Golan, and Bradley D. Stein, California Colleges and Universities Collaborate to Support Student Mental Health, Santa Monica, Calif.: RAND Corporation, RR-689-CMHSA, 2014b. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR689.html

Woodbridge, Michelle W., Jennifer Yu, Asha Goldweber, Shari Golan, and Bradley D. Stein, California K–12 School and Community Collaborations: Facilitators, Challenges, and Impact on Student Mental Health, Santa Monica, Calif.: RAND Corporation, RR-956-CMHSA, 2015a. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR956.html

Woodbridge, Michelle W., Jennifer Yu, Asha Goldweber, Shari Golan, and Bradley D. Stein, California College and University Collaborations: Facilitators, Challenges, and Impact on Student Mental Health, Santa Monica, Calif.: RAND Corporation, RR-955-CMHSA, 2015b. As of January 26, 2015:
http://www.rand.org/pubs/research_reports/RR955.html

This research was conducted in RAND Health, a division of the RAND Corporation.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.