Evaluation of the California Mental Health Services Authority's Prevention and Early Intervention Initiatives

In 2004, California voters passed Proposition 63, the Mental Health Services Act (MHSA), which includes a mandate that the state provide prevention and early intervention (PEI) services and education for people who experience mental illness in the state of California. The California Mental Health Services Authority (CalMHSA), a coalition of California counties formed to provide economic and administrative support to mental health service delivery, formed the Statewide PEI Implementation Program based on extensive recommendations from a large number of stakeholders statewide. The CalMHSA Statewide PEI program is composed of three strategic initiatives focusing on: (1) reduction of stigma and discrimination towards those with mental illness, (2) prevention of suicide, and (3) improvement in student mental health. Each initiative is implemented with the assistance of community agencies serving as PEI Program Partners (see Table 1 for the Program Partners under each initiative).

In 2011, the RAND Corporation was contracted by CalMHSA to design and implement a three-year statewide evaluation of the three CalMHSA PEI Program initiatives, the overall effort, and specific Program Partner activities. The ongoing evaluation has three aims:

• To evaluate individual Program Partners' progress toward meeting statewide goals and objectives
• To assess the resources and capacities developed by Program Partners as part of their CalMHSA-sponsored efforts, including the structural and operational processes that define the resources and capacities
• To evaluate outcomes of Program Partners' activities, both short- and long-term.

To meet the aims set forth above, the evaluation focuses on evaluating Program Partner resource and capacity-building efforts that fall into six types of core activities that occur across initiatives:

1. the development of policies, protocols, and procedures
2. networking and collaboration
3. informational resources
4. training and educational programs
5. media/social marketing campaigns and interventions to influence how media productions depict mental health
6. hotline and “warmline” operations, that is, providing crisis support and basic social support, respectively.

Because Program Partners are required to conduct evaluations of their activities, RAND is evaluating a strategically selected subset of activities identified through conversations with CalMHSA and the Program Partners.

In addition to evaluating the activities above, the RAND evaluation is also developing baseline assessments of population risk factors and outcomes for the initiatives. These baseline assessments provide a platform for longer term monitoring of population risk factors and outcomes over time. Current baseline population tracking includes an analysis of county- and region-wide suicide rates, an in-progress student- and faculty survey of the school mental health climate across California, and a statewide survey of California adults' beliefs about suicide, mental health stigma and discrimination, and the mental health climate in schools.

The evaluation aims are derived from the priorities set forth in the CalMHSA Statewide PEI Implementation Work Plan* and are set forth in detail in an evaluation plan developed by RAND and approved by CalMHSA. In addition, the RAND evaluation team has been providing technical assistance to Program Partners to promote the development of their capability to assist in the evaluation of the initiatives and promote continuous quality improvement efforts.

This article provides a summary on a more detailed interim report of the RAND evaluation of the CalMHSA Statewide PEI Program. This detailed interim report presents early findings on the capacities and resources developed by the Stigma and Discrimination Reduction, Suicide Prevention, and Student Mental Health initiatives. In addition, results of a baseline, statewide survey of the general population of California's knowledge, attitudes, and beliefs toward mental health are presented. Summaries of each of these topics are provided below, and more detail will be available in the forthcoming interim report. While many programmatic activities have been implemented within the past year, others activities are still in development and will be implemented over the coming year. Thus, results presented at this time in this summary document and in the detailed interim report are necessarily preliminary.

Stigma and Discrimination Reduction

The RAND evaluation assesses select activities of Stigma and Discrimination Reduction (SDR) Initiative Program Partners that fall into four of the six core activity areas (see Table 2).

Stigma and Discrimination Reduction Initiative Program Partners have developed many capacities and resources related to improving policies, procedures, and protocols, as well as informational resources related to stigma and discrimination reduction. These capacities and resources include fact sheets, toolkits, and reviews that identify and assess promising practices in SDR in community organizations. Many online resources have also been developed. For online resources, we present early results from website analytics to track how users are finding and interacting with Program Partner sites, what resources they are downloading, and where in California site visitors are located. These results show that there have been over 45,000 visits to online resources sponsored by CalMHSA, and site visitors have come from many areas across California. Continued tracking of online dissemination and implementation of tools for tracking reach off line are ongoing. Tools for understanding resource effectiveness are also being implemented.

SDR Program Partners are also making available a host of trainings and educational programs. These offerings include trainings for a wide variety of audiences, such as people with mental health challenges, family members of people with mental health challenges, landlords, health providers, county mental/behavioral health service managers, teachers, and students. Many of these trainings utilize contact with consumers of mental health services to help reduce stigma and discrimination (an evidence-based practice). Because tools for tracking the reach of the trainings and educational presentations and their impact on attitudes, beliefs, and behaviors have only been in place for a short time, we are unable to report results on reach and attitude change at this time.

In addition to providing informational resources and trainings, SDR Program Partners are implementing two media-related stigma and discrimination reduction strategies: providing media training to journalism and entertainment professionals and conducting a social marketing campaign targeting populations across the lifespan, with an emphasis on youth. Evaluations of these activities are in progress, and no results are available at this time.

Suicide Prevention

The RAND evaluation of the Suicide Prevention (SP) Initiative focuses on the evaluation of select Program Partner efforts that focus on four of the six core activity areas (see Table 3).

The Suicide Prevention (SP) Initiative Program Partners are focused on building hotline and “warmline”** capacities across the state, promoting networking and collaboration among hotlines and “warmlines,” and using social marketing efforts to promote suicide and mental health awareness. The evaluation includes an assessment of the networking and collaboration resulting from the efforts of Program Partner Didi Hirsch (a mental health service agency with a dozen locations in and around Los Angeles), which is facilitating the California Suicide Prevention Network (CSPN). Reviews of related documents (e.g., Memoranda of Understanding [MOUs] with partners and emergency/crisis intervention protocols, policy recommendations, and meeting rosters and agendas) are in progress, and RAND will conduct key informant interviews and a collaboration survey at a later stage of the evaluation.

Four new crisis hotlines have been created, and several existing hotlines are seeking accreditation or have been accredited since the beginning of the contract period. To understand the reach of hotline and warmline operations, we are tracking call volume. We have developed a protocol for systematically monitoring hotline call quality.

Evaluations of several suicide intervention trainings (LivingWorks' SafeTalk and ASIST trainings) are ongoing. Data on the demographics of training participants reached to date are available, and post-training surveys indicate high satisfaction with the trainings and increases in perceptions of self-efficacy and intentions to help people at risk. Monitoring of fidelity to the ASIST training protocol is in progress.

One Program Partner, AdEase, is conducting a social marketing campaign. The evaluation of SP social marketing activities is still in progress. We will evaluate campaign messages and their efficacy and assess the campaign's reach at a later point.

In addition to the evaluation of the key Program Partner activities above, we have analyzed suicide fatalities in California to establish baselines against which later suicide rates may be compared. Age-adjusted suicide rates by region are presented in Figure 1. Two major findings emerge from this analysis. First, the suicide rate is highest in California's most-rural areas (e.g., Humboldt, Mendocino, Siskiyou, Butte, and Amador counties), indicating that those who live in these areas are at higher risk for suicide. Second, suicides in these areas actually account for a very small proportion of California's overall number of suicides (approximately 6%), indicating that resources must still be allocated to the areas of the state with the highest numbers of suicides.

Student Mental Health

The RAND evaluation of the Student Mental Health (SMH) Initiative assesses select Program Partners' efforts in the three core activity areas highlighted in Table 4. The Student Mental Health (SMH) Initiative Program Partners are focusing on improving the mental health of both K-12 and Higher Education students throughout California. These Program Partners are developing resources for improving student mental health, conducting trainings for educational professionals, and promoting networking and collaboration among school campuses and neighboring community organizations.

The evaluation of SMH activities related to networking and collaboration will focus on the California County Superintendents' Educational Services Association (CCSESA) county consortia, the State SMH Policy Workgroup, University of California (UC) and California State University (CSU) SMH Initiative Advisory Groups, California Community Colleges (CCC) Regional Strategizing Forums, and inter- and intra-campus collaborations among the higher-education Program Partners. Reviews of related documents (e.g., meeting rosters, agendas, policy recommendations) are in progress. Key informant interviews and a collaboration survey will be conducted later.

SMH Program Partners are making many informational resources available online. These include resources about mental health issues for students and information for faculty and staff regarding approaches to supporting students with mental health needs. Thus far, the websites hosting informational resources have been reviewed by RAND evaluators for content and target audience. Website analytics and feedback survey data are currently available for online resources developed by CCSESA (for K-12 schools). Early results are presented in this report and indicate that initial interest in the website has come primarily from school administrators and mental health professionals who are interested in students of all ages. Site visitors reported coming to the site to seek materials on a wide variety of topics, with mental health/wellness, bullying, and behavior management among the most prevalent. We are currently developing a follow-back survey designed to assess the usefulness of the materials.

SMH Program Partners implemented a variety of training programs to promote the early identification and appropriate referral of students experiencing mental health issues. Thus far, we have provided technical assistance to SMH Program Partners to implement training surveys, as well as tools for tracking the reach of trainings. In the future, several trainings will be selected for detailed content analysis. We present available data on training presentations and their reach in the report. Preliminary analyses of training survey data indicate that participants reported being satisfied with the training and experienced increased self-efficacy and behavioral intentions after undergoing training.

In addition to the evaluation of the key Program Partner activities above, we have designed baseline surveys of student, faculty, and staff perceptions of school climate and student attitudes and behavior related to mental health, and we are in the process of collecting data for them. The K-12 survey has not yet been fielded, but preliminary data based on 6,309 higher education students and 3,025 faculty and staff are available. Based on their responses, about 20% of higher education students are likely experiencing a mental health problem, and 25% of student respondents reported either having used or having been referred to campus mental health services. Some 25% to 35% of students reported that their academic performance was negatively affected by anxiety or depression. However, 67% of students indicated that they know where to go for help with a personal problem. Students generally believed that the campus climate with respect to mental health issues is positive (e.g., more friendly than hostile). Faculty and staff agreed that their campuses provide adequate mental health counseling and support to students. 24% of faculty and staff reported having talked with a student about mental health once or twice, and 46% did not discuss mental health with students in the past month. Twenty percent of faculty/staff report having attended some form of training on student mental health. Over 50% of faculty/staff stated that they knew where to refer students who need mental health resources.

In summary, SMH Program Partners are engaging in a wide variety of activities, including collaborating with other organizations, providing informational resources, and offering training on student mental health issues. RAND evaluation activities designed to assess reach of these expanded capacities and resources are in progress. The ongoing administration of surveys of SMH climate provides a useful baseline against which to compare future school climate data.

General Population Survey: Baseline Preliminary Results

We conducted a general population statewide survey of California adults. The survey included questions about such topics as mental health literacy, stigmatizing attitudes, and exposure to CalMHSA PEI efforts. The main purpose of the survey is to serve as a baseline against which later data on the topics above can be compared. It also serves as a measure of early exposure of the general population to CalMHSA activities. A similar survey will be fielded in approximately one year so changes from baseline can be determined. We caution that one year is a short timeframe in which to observe widespread population-level change and suggest continued tracking to observe population-level change over time.

In conducting the general population survey, we aimed to establish baseline levels of knowledge, attitudes, and beliefs about SP, SDR, and SMH among the California adult population and to learn about early exposure to CalMHSA activities. Results presented here are preliminary, and we are continuing to analyze the survey data. We reached a diverse group of 2,001 California adults (age 18 and over). The sample closely matches known California population characteristics in terms of sex, age, race, ethnicity, education, income, and employment.

Two-thirds of respondents were aware of stigma and discrimination toward people with mental health challenges. They personally held some stigmatizing attitudes and beliefs (e.g., about one quarter of respondents thought that people with mental health challenges are dangerous), but many also reported some positive beliefs about potential for recovery and contributing positively to society (e.g., 70% of respondents thought that a person with mental illness can recover). 92% of respondents expressed a willingness to support people with mental health challenges. 20% of respondents reported that they would hesitate to disclose having experienced a mental health challenge to their friends or family, and 17% indicated that they would hesitate to seek treatment for such a challenge out of fear of what others would think.

Respondents varied in their opinions about suicide. About two-thirds of respondents recognized that suicide is preventable, and just over half thought that suicide is always preceded by warning signs. About half also believed, incorrectly, that talking about suicide can cause suicide. Nearly half of respondents did not know that men are at greater risk of completing suicide than women. Respondents indicated that if they were having suicidal thoughts, they would be more likely to seek face-to-face help from a counselor or other mental health professional than to use other possible resources.

Respondents with a child in a K-12 school or in an institution of higher education and respondents who were themselves students in an institution of higher education were asked about school climate for handling issues related to mental health. Parents of K-12 students and students in higher educational institutions indicated that they “somewhat agree” with the idea that their school helped students and provided quality counseling and other resources to help students with social, emotional, and behavioral problems. Respondents who were themselves students typically agreed that their institution helps students and provides quality counseling.

Exposure to CalMHSA activities at the population level has been difficult to detect early in the project period. Eleven percent of respondents reported having seen or heard of the slogan “Each Mind Matters,” 8% had heard of “Reach Out,” and 9% had seen or heard of “Suicide is Preventable.” However, 2% or less of respondents visited the Each Mind Matters, Reach Out, or Suicide is Preventable websites. We note, however, that the Each Mind Matters website did not exist until partway through the data collection period. Also, some social marketing activities were targeted toward 14- to 24-year-olds, and the survey was only administered to Californians 18 and older. Some 39% of respondents reported seeing or hearing ads with specific AdEase taglines (e.g., “Know the Signs”). Furthermore, 16% reported having attended some sort of training about mental illness, but we cannot determine if these trainings were among those implemented through CalMHSA's PEI initiatives.

Notes

* This document is available online at http://calmhsa.org/programs/pei-statewide-projects/.

** A warmline is a non-crisis telephone service that provides encouragement and support to persons in need.

The research described in this article was sponsored by the California Mental Health Services Authority (CalMHSA), and was produced within RAND Health, a division of the RAND Corporation.