Evaluation of the California Mental Health Services Authority's Prevention and Early Intervention Initiatives

Progress and Preliminary Findings

by M. Audrey Burnam, Sandra H. Berry, Jennifer L. Cerully, Nicole K. Eberhart

This Article

RAND Health Quarterly, 2014; 4(3):12

Abstract

The California Mental Health Services Authority (CalMHSA) statewide Prevention and Early Intervention (PEI) program comprises three strategic initiatives: (1) reduction of stigma and discrimination toward those with mental illness, (2) prevention of suicide, and (3) improvement in student mental health. Community agencies serve as PEI program partners to perform activities intended to meet the goals of the initiatives. This article evaluates the progress of the PEI program partners in achieving their goals and establishes baseline population tracking of key risk factors and long–term outcomes targeted by the initiatives. Based on a model to assess the program partners' capacities and resources and a recent survey of California adults, this article shows that the partners have greatly expanded their abilities to launch numerous PEI activities and programs.

For more information, see RAND RR-438-CMHSA at https://www.rand.org/pubs/research_reports/RR438.html

Full Text

In 2004, California voters passed Proposition 63, the Mental Health Services Act (MHSA), which includes a mandate that the state provide prevention and early intervention (PEI) services and education for people who experience mental illness in the state of California. The California Mental Health Services Authority (CalMHSA), a coalition of California counties formed to provide economic and administrative support to mental health service delivery, formed the Statewide PEI Implementation Program based on extensive recommendations from a large number of stakeholders statewide. The CalMHSA Statewide PEI program is composed of three strategic initiatives focusing on: (1) reduction of stigma and discrimination toward those with mental illness, (2) prevention of suicide, and (3) improvement in student mental health. Each initiative is implemented with the assistance of community agencies serving as PEI Program Partners (see Table 1 for the Program Partners under each initiative).

Table 1. CalMHSA Statewide PEI Program Partners by Initiative

Stigma and Discrimination Reduction (SDR) Suicide Prevention (SP) Student Mental Health (SMH)

Disability Rights California

Entertainment Industries Council, Inc.

Integrated Behavioral Health Project/Center for Care Innovations

Mental Health America of California

Mental Health Association of San Francisco

National Alliance on Mental Illness

Runyon, Saltzman & Einhorn

United Advocates for Children and Families

SDR Consortium

AdEase

Didi Hirsch Psychiatric Services

Family Service Agency of the Central Coast

Family Service Agency of Marin

Institute on Aging Center

Kings View

LivingWorks

San Francisco Suicide Prevention

Transitions Mental Health Association

California County Superintendents Educational Services Association

California Department of Education

California Community Colleges

California State University

University of California

In 2011, the RAND Corporation was contracted by CalMHSA to design and implement a three-year statewide evaluation of the statewide PEI initiative. The evaluation involves assessing PEI Program Partner activities, the three major initiatives (i.e., stigma and discrimination reduction [SDR], suicide prevention [SP], and student mental health [SMH] initiatives), and the overall CalMHSA statewide PEI initiative. Specifically, the RAND evaluation team is collaborating with the PEI Program Partners to achieve the following evaluation aims:

  • Evaluate PEI Program Partners' progress toward meeting statewide objectives
  • Assess the activities implemented and resources created by PEI Program Partners
  • Evaluate program outcomes, including:
  • Targeted program capacities and their reach (i.e., the number of people exposed to program materials, services, social marketing messages, and training)
  • Short-term outcomes (e.g., attitudes and knowledge about mental illness, behavior toward people with mental illness)
  • Longer-term outcomes (e.g., reduced suicide, reduced discrimination, improved student performance).

To meet these aims, the evaluation focuses on evaluating Program Partner resources and capacity-building efforts. We have organized these into six types of core activities that occur across initiatives:

  1. the development of policies, protocols, and procedures
  2. networking and collaboration
  3. informational resources
  4. training and educational programs
  5. media/social marketing campaigns and interventions to influence how media productions depict mental health
  6. hotline and “warmline” operations, that is, providing crisis support and basic social support, respectively.

Because Program Partners are required to conduct evaluations of their activities, RAND is evaluating a strategically selected subset of activities identified through conversations with CalMHSA and the Program Partners.

In addition to evaluating these activities, the RAND evaluation is also developing baseline assessments of population risk factors and outcomes for the initiatives. These baseline assessments provide a platform for longer-term monitoring of population risk factors and outcomes over time. The evaluations' baseline population tracking includes an analysis of county- and region-wide suicide rates, an in-progress student and faculty survey of the school mental health climate across California, and a statewide survey of California adults' beliefs about suicide, mental health stigma and discrimination, and the mental health climate in schools.

The evaluation aims are derived from the priorities set forth in the CalMHSA Statewide PEI Implementation Work Plan* and are set forth in detail in an evaluation plan developed by RAND and approved by CalMHSA. In addition, the RAND evaluation team has been providing technical assistance to Program Partners to enhance their ability to assist in the evaluation of the initiatives and promote continuous quality improvement efforts.

This study presents early findings on the capacities and resources developed by the Stigma and Discrimination Reduction, Suicide Prevention, and Student Mental Health initiatives. In addition, results of a baseline statewide survey of the general population of California's knowledge, attitudes, and beliefs toward mental health are presented. While Program Partners implemented many activities within the past year, other activities are still in development and will be implemented over the coming year. Thus, results presented at this time are necessarily preliminary.

Stigma and Discrimination Reduction

Within the categories of core activities identified above, the RAND evaluation strategically focused on the central and well-defined activities that represent major program efforts. In the Stigma and Discrimination Reduction (SDR) Initiative, the RAND evaluation identified and focused on key activities that fell under four of the six core activity areas (see Table 2).

Table 2. Key Activities Being Evaluated Under the Stigma and Discrimination Reduction Initiative

Type of Core Activity
Development of policies, protocols, and procedures X
Networking and collaboration
Development of informational/online resources X
Training and educational programs X
Media/social marketing campaigns and interventions to influence media production X
Hotline and “warmline” operations

SDR Initiative Program Partners have developed many capacities and resources, including improving policies, procedures, and protocols, as well as informational resources related to stigma and discrimination reduction. These capacities and resources include fact sheets, tool kits, and reviews that identify and assess promising practices in SDR in community organizations. Many online resources have also been developed. For online resources, we present early results from website analytics to track how users are finding and interacting with Program Partner sites, what resources they are downloading, and where in California site visitors are located. These results show that there have been over 45,000 visits to online resources sponsored by CalMHSA, and site visitors have come from many areas across California. We continue to track the Program Partners' online dissemination of tools and materials in order to assess program reach. We are also implementing tools for understanding resource effectiveness.

SDR Program Partners are also hosting trainings and educational programs. These offerings target a wide variety of audiences, such as people with mental health challenges, family members of people with mental health challenges, landlords, health providers, county mental/behavioral health service managers, teachers, and students. Many of these trainings utilize contact with consumers of mental health services to help reduce stigma and discrimination (an evidence-based practice). Because tools for tracking the reach and impact of these trainings and presentations have been in place only for a short time, we are as yet unable to report results on these.

In addition to providing informational resources and trainings, SDR Program Partners are implementing two media-related stigma and discrimination reduction strategies: providing media training to journalism and entertainment professionals and conducting a social marketing campaign targeting populations across the lifespan, with an emphasis on youth. Evaluations of these activities are in progress; no results are available at this time.

Suicide Prevention

The RAND evaluation of the Suicide Prevention (SP) Initiative determined that SP Initiative Program Partners' central, well-defined activities fall into four of the six core activity areas, which represent the major program efforts (see Table 3).

Table 3. Key Activities Being Evaluated Under the Suicide Prevention Initiative

Type of Core Activity
Development of policies, protocols, and procedures
Networking and collaboration X
Development of informational/online resources
Training and educational programs X
Media/social marketing campaigns and interventions to influence media production X
Hotline and “warmline” operations X

The SP Initiative Program Partners are focused on building hotline and “warmline”** capacities across the state, promoting networking and collaboration among hotlines and “warmlines,” and using social marketing efforts to promote suicide and mental health awareness. Our evaluation includes an assessment of the networking and collaboration resulting from the efforts of Program Partner Didi Hirsch (a mental health service agency with a dozen locations in and around Los Angeles), which is facilitating the California Suicide Prevention Network (CSPN). Reviews of related documents (e.g., Memoranda of Understanding [MOUs] with partners and emergency/crisis intervention protocols, policy recommendations, and meeting rosters and agendas) are in progress, and RAND will conduct key informant interviews and a survey regarding collaboration at a later stage of the evaluation.

Program Partners created or expanded four new crisis response services, and several existing hotlines are seeking accreditation or have been accredited since the beginning of the contract period. To understand the reach of hotline and warmline operations, we are tracking call volume. We have developed a protocol for systematically monitoring hotline call quality.

Evaluations of several suicide intervention trainings (LivingWorks' SafeTalk and ASIST trainings) are ongoing. Data on the demographics of training participants reached to date are available, and post-training surveys indicate high satisfaction with the trainings and increases in perceptions of self-efficacy and intentions to help people at risk. Monitoring of fidelity to the ASIST training protocol is in progress.

One Program Partner, AdEase, is conducting a social marketing campaign related to suicide prevention. The evaluation of SP social marketing activities is still in progress. Campaign components are described in the study; we will evaluate campaign messages and their efficacy during years two and three of the evaluation. Preliminary data on the reach of the Know the Signs website (www.suicideispreventable.org) are presented in this report and show that over 470,000 visits were made to the site between November 2012 and February 2013. More data on campaign reach will be provided at a later point.

In addition to the evaluation of the key Program Partner activities above, we have analyzed suicide fatalities in California to establish baselines against which later suicide rates may be compared. Age-adjusted suicide rates by region are presented in Figure 1. Two major findings emerge from this analysis. First, the suicide rate is highest in California's most-rural areas (e.g., Humboldt, Mendocino, Siskiyou, Butte, and Amador counties), indicating that those who live in these areas are at higher risk for suicide. Second, suicides in these areas actually account for a very small proportion of California's overall number of suicides (approximately 6%), indicating that resources must still be allocated to the areas of the state with the highest numbers of suicides.

Figure 1. Map of Age-Adjusted Suicide Rates by Region (2008–2010)

Figure 1. Map of Age-Adjusted Suicide Rates by Region (2008–2010)

Student Mental Health

The RAND evaluation of the Student Mental Health (SMH) Initiative strategically assesses Program Partners' most central efforts, which fall into the three core activity areas highlighted in Table 4. The SMH Initiative Program Partners are focusing on improving the mental health of both K–12 and higher education students throughout California. These Program Partners are developing resources for improving student mental health, conducting trainings for educational professionals, and promoting networking and collaboration among school campuses and neighboring community organizations.

Table 4. Projected Workforce and Hospital Bed Needs—Base Case

Type of Core Activity
Development of policies, protocols, and procedures  
Networking and collaboration X
Development of informational/online resources X
Training and educational programs X
Media/social marketing campaigns and interventions to influence media production  
Hotline and “warmline” operations  

The evaluation of SMH activities related to networking and collaboration will focus on the California County Superintendents Educational Services Association county consortia, the State SMH Policy Workgroup, University of California and California State University SMH Initiative Advisory Groups, California Community Colleges Regional Strategizing Forums, and inter- and intra-campus collaborations among the higher-education Program Partners. Reviews of related documents (e.g., meeting rosters, agendas, policy recommendations) are in progress. Key informant interviews and a collaboration survey will be conducted later.

SMH Program Partners are making many informational resources available online. These include resources about mental health issues for students and information for faculty and staff regarding approaches to supporting students with mental health needs. Thus far, RAND evaluators have reviewed websites hosting informational resources, for content and target audience. Website analytics and feedback survey data are currently available for online resources developed by California County Superintendents Educational Services Association (for K–12 schools). Early results are presented in this report and indicate that initial interest in the website has come primarily from school administrators and mental health professionals who are interested in students of all ages. Site visitors reported coming to the site to seek materials on a wide variety of topics, with mental health/wellness, bullying, and behavior management among the most prevalent. We are currently developing a follow-back survey to assess the usefulness of the materials.

SMH Program Partners implemented a variety of training programs to promote the early identification and appropriate referral of students experiencing mental health issues. Thus far, we have provided technical assistance to SMH Program Partners to implement tools to evaluate SMH trainings, as well as tools for tracking the reach of trainings. In the future, several trainings will be selected for detailed content analysis. We present available data on training presentations and their reach in this report. Preliminary analyses of training survey data indicate that participants reported being satisfied with the training and experienced increased self-efficacy and behavioral intentions after undergoing training.

In addition to the evaluation of the key Program Partner activities above, we have designed baseline surveys of student, faculty, and staff perceptions of school climate and student attitudes and behavior related to mental health, and we are in the process of collecting these data. The K–12 survey has not yet been fielded, but preliminary data based on 6,309 higher education students and 3,025 faculty and staff are available. Their responses suggest that about 20 percent of higher education students are likely experiencing a mental health problem, and 25 percent of student respondents reported either having used or having been referred to campus mental health services. Some 25 to 35 percent of students reported that their academic performance was negatively affected by anxiety or depression. However, 67 percent of students indicated that they know where to go for help with a personal problem. Students generally believed that the campus climate with respect to mental health issues is positive (e.g., more friendly than hostile). Faculty and staff agreed that their campuses provide adequate mental health counseling and support to students. Twenty-four percent of faculty and staff reported having talked with a student about mental health once or twice, 30 and 46 percent did so a few or many times, but almost half (46%) did not discuss mental health with students in the past six months. Twenty percent of faculty/staff reported having attended some form of training on student mental health during the past six months. Over 50 percent of faculty/staff stated that they knew where to refer students who need mental health resources.

In summary, SMH Program Partners are engaging in a wide variety of activities, including collaborating with other organizations, providing informational resources, and offering training on student mental health issues. RAND evaluation activities designed to assess reach of these expanded capacities and resources are in progress. The ongoing administration of surveys of SMH climate provides a useful baseline against which to compare future school climate data.

General Population Survey: Baseline Preliminary Results

We used random digit dialing to conduct a general population statewide survey of California adults. The survey includes questions about such topics as mental health literacy, stigmatizing attitudes, and exposure to CalMHSA PEI efforts. The main purpose of the survey is to serve as a baseline against which later data on these topics can be compared. It also serves as a measure of early exposure of the general population to CalMHSA activities. A similar survey will be fielded in approximately one year so that changes from baseline can be determined. We caution that one year is a short time frame in which to observe widespread population-level change and suggest continued tracking to observe population-level change over time.

Results presented here are preliminary, and we are continuing to analyze the survey data. We reached a diverse group of 2,001 California adults (age 18 and over). The sample closely matches known California population characteristics in terms of sex, age, race, ethnicity, education, income, and employment.

Two-thirds of respondents were aware of stigma and discrimination toward people with mental health challenges. Some respondents indicated stigmatizing attitudes and beliefs (e.g., about one-quarter of respondents thought that people with mental health challenges are dangerous), but many also reported some positive beliefs about potential for recovery and contributing positively to society (e.g., 70 percent of respondents thought that a person with mental illness can recover). Some 92 percent of respondents expressed a willingness to support people with mental health challenges. About 20 percent of respondents reported that they would hesitate to disclose having experienced a mental health challenge to their friends or family, and 17 percent indicated that they would hesitate to seek treatment for such a challenge out of fear of what others would think.

Respondents varied in their opinions about suicide. About two-thirds of respondents believed that suicide is preventable, and just over half thought that suicide is always preceded by warning signs. About half also believed, incorrectly, that talking about suicide can cause suicide. Nearly half of respondents did not know that men are at greater risk of completing suicide than women. Respondents indicated that if they were having suicidal thoughts they would be more likely to seek face-to-face help from a counselor or other mental health professional than to use other possible resources.

Respondents with a child in a K–12 school or in an institution of higher education and respondents who were themselves students in an institution of higher education were asked about school climate for handling issues related to mental health. Parents of K–12 students and students in higher educational institutions indicated that they “somewhat agree” with the idea that their school helped students and provided quality counseling and other resources to help students with social, emotional, and behavioral problems. Students typically agreed that their institution helps students and provides quality counseling.

Exposure to CalMHSA activities at the population level has been difficult to detect early in the project period. Eleven percent of respondents reported having seen or heard of the slogan “Each Mind Matters,” 8 percent had heard of “ReachOut,” and 9 percent had seen or heard of “Suicide Is Preventable.” However, 2 percent or less of respondents visited the Each Mind Matters, ReachOut, or Suicide Is Preventable websites. We note, however, that the Each Mind Matters website did not exist until partway through the data collection period. Also, some social marketing activities were targeted toward 14- to 24-year-olds, and the survey was only administered to Californians 18 and older. Thirty-nine percent of respondents reported seeing or hearing ads with specific AdEase taglines (e.g., “Know the Signs”). Furthermore, 16 percent reported having attended some sort of training about mental illness, but we cannot determine if these trainings were among those implemented through CalMHSA's PEI initiatives.

Commentary

This article presents early evaluation findings for many newly developed program activities that together represent the implementation of interdependent statewide strategies designed to reduce mental health stigma and discrimination, prevent suicide, and improve student mental health. Many program activities are not yet fully implemented and the evaluation is ongoing. Nonetheless, the question of whether these programs are producing their intended effects is a pressing one for California decisionmakers and other stakeholders. This commentary offers our perspective on how well it is progressing so far.

There is a logical, science-informed path from the statewide strategic plan to achieving actual reduction in mental health stigma and discrimination, reduction in suicide, and improvement in student mental health. This path involves: (1) the strategic planning of comprehensive, inter-related program components, (2) development of new PEI program capacities, (3) delivery of new program activities to achieve broad reach to California's diverse population and result in significant exposure to program materials, (4) impact of program activities on targeted short-term outcomes such as knowledge and attitudes, and (5) impact on longer-term outcomes for California's population.

These PEI initiatives are bold and ambitious efforts for the state of California—both because of the uniqueness of a new strategic “statewide” approach to prevention and early intervention programs and because they are managed by a relatively new and innovative organizational body that involves joint decisionmaking across California's many and diverse counties. The components of the statewide PEI strategic plan were carefully and broadly informed through a strategic planning process that involved diverse stakeholders.

To date, it is clear that Program Partners have been highly productive in developing new program capacities that relate to the components of the strategic plan. Furthermore, the launching of many program activities is well under way. This is impressive given the relatively short time Program Partners have had to develop and implement new program activities. So far, reach of program activities is relatively limited (a result of being in the early stages of implementation) or cannot yet be determined, and many program activities are in a phase of rapid expansion of their reach. We do not know yet whether programs are having their intended short-term impacts on participants/audiences, but we expect to be able to answer those questions for key program activities over the next one to two years, within the time frame of this evaluation. We caution that it may be unrealistic to expect observable population changes in the long-term outcomes of interest during this period, given the start-up time required to build and launch new programs, the relatively brief period over which program effects will be observed, and the importance of broad population reach and exposure for prevention to have an impact.

Notes

* This document is available online at http://calmhsa.org/programs/pei-statewide-projects/.

** A warmline is a non-crisis telephone service that provides encouragement and support to persons in need.

The research described in this article was sponsored by the California Mental Health Services Authority (CalMHSA), and was produced within RAND Health, a division of the RAND Corporation.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.