Effects of the Integrated Behavioral Health Project's Efforts to Promote Integrated Care Under Funding from the California Mental Health Services Authority

Key Findings

• IBHP effectively reached its target audience of health care providers and mental health services providers.
• Many respondents shared the information they obtained with their own colleagues, other organizations, and with people experiencing mental illness or their family members, expanding this reach.
• About one in five respondents reported making policy or practice changes as a result of receiving information or services from IBHP.
• A similar number reported developing new collaborative relationships with other organizations.

Integrated care—that is, the care that “occurs when mental health specialty and general medical care providers work together to address both the physical and mental health needs of their patients” (Butler et al., 2008)—has been recommended as a strategy to address a major public health problem: the high rate of detrimental physical health conditions and shorter lifespans among people with mental illness (Colton et al., 2006; Druss et al., 2011; Institute of Medicine, 2006; Koyanagi, 2004; President's New Freedom Commission on Mental Health, 2003; Unützer et al., 2006). Integrating mental and physical health care may also increase the number of individuals who receive needed mental health treatment by providing mental health care in primary care settings, a less-stigmatizing venue than mental health specialty care settings (U.S. Department of Health and Human Services, 1999; Shim and Rust, 2013; World Health Organization and World Organization of Family Doctors, 2008).

The California Mental Health Services Authority (CalMHSA), an organization of county governments working to improve mental health outcomes for individuals, families, and communities, funded the Integrated Behavioral Health Project (IBHP) to develop resources and work in partnership with state and county organizations to provide information and services promoting integrated care. IBHP was funded under the Values, Practices, and Policies Promoting Integrated Health program within CalMHSA's stigma and discrimination reduction (SDR) initiative.^[1] The objectives of the program were, in part, to address mental illness stigma and discrimination among care providers and encourage integration of the health systems that serve people with mental illness (Clark et al., 2013).

This article describes the methods and results of a RAND survey of a sample of recipients of IBHP information and services under the SDR initiative. IBHP reports administering the survey to the following individuals: recipients of an integrated care toolkit, participants in annual summits on integrated care in California, and attendees of webinars on topics related to integrated care.^[2] The integrated care toolkit (IBHP, 2013) is intended to support collaborations across primary care, mental health, and substance abuse treatment sectors, and thus is intended for stakeholders working in those fields. The toolkit contains many different types of information, including scientific research on integrated behavioral health, quotes from medical and behavioral health providers on the positive benefits of integrated care, illustrative examples from case studies, practical information on the logistics of establishing interagency collaborations (e.g., a sample memorandum of understanding, screening instruments, job descriptions), and a section on why mental health stigma should matter to health care providers. Annual California Innovations Summits, held in 2013 and 2014,^[3] each spanned two days and were designed for leaders from county mental health and substance use disorder services, provider organizations, federally qualified health centers, safety net health plans, state agency leaders and staff, and consumer and family organizations. The purpose was to educate and share information about innovations in integrating and coordinating care for persons with complex mental health and substance use disorders as well as medical conditions. IBHP conducted virtual technical assistance and education webinars on various topics related to integrated care, such as mental health stigma and integrated care, integrated care for ethnic minorities, payment models to sustain integration, and key steps to integrating practices. These webinars targeted key stakeholders in integrated care, such as medical and mental health clinic leadership, researchers, consultants, health plans, and health professionals, and many of them are available online at the Farley Health Policy Center YouTube channel (“Farley Health Policy Center,” undated).

Methods

To assess the ways in which participants in IBHP's CalMHSA-funded efforts used the information and services they received, we developed an online follow-up survey. This survey was to be administered approximately six months after participation in IBHP activities. IBHP sent an email invitation to individuals who received its integrated care toolkit, participants in annual summits on integrated care in California, and attendees of webinars on topics related to integrated care. This email asked participants to complete a follow-up survey and provided a web link to the online survey. In an effort to maximize participant responses, IBHP staff sent email invitations in May and December 2014. A total of 1,518 survey invitations were sent; some addresses were nonfunctioning, resulting in a total of 1,369 issued invitations (90 percent of the total number of email addresses collected).

Measures

Respondents were asked what type of information or services they received from IBHP. They were provided with the following response options: attended an in-person event, training, or educational presentation; attended an online event, training, or educational presentation; received one-on-one technical assistance; received materials or information (online, in person, or by mail/email); visited the IBHP website; or none of the above. Participants could select more than one response if appropriate.

Respondents reported whether and how they had used the information or services received from IBHP. They did so by endorsing statements describing uses following from our logic model describing the intended outcomes of SDR program partners' efforts, including those of IBHP. Sample statements included “I acted in ways that are more supportive of people with mental illness” and “I/My organization started a new collaborative relationship with another organization” (see Appendix A for the full list of statements). Statements covered possible uses in several domains, including information sharing, engaging in more-supportive behavior, and making organizational or policy changes. Respondents could also indicate that they used the information “in another way” or that they did not use it. Respondents also reported their likelihood of using the information or services they obtained in the future.

CalMHSA emphasizes the use of culturally appropriate and competent approaches to prevention and early intervention in order to improve mental health among all members of California's diverse population (Clark et al., 2013). To assess the cultural appropriateness of IBHP's efforts, respondents indicated whether they would recommend the training, presentation, or information to someone of their cultural background, as well as their level of agreement with the following statement: “The information or resources was sensitive to my cultural background (race, religion, language, sexual orientation, etc.).” To assess the degree to which the information or services provided by IBHP fostered culturally competent approaches, respondents indicated their level of agreement with the following statement: “The information taught me how to be culturally sensitive when talking with someone with a mental illness.” Respondents could also endorse the following item when asked whether and how they used the information or services they received: “I/My organization adopted more culturally competent approaches to reducing stigma and discrimination against people with mental illness.”

To help us understand program reach and whether outcomes differed by various subgroups of program participants, respondents were asked several demographic questions and whether they served in any of a list of stakeholder roles that potentially put them in a position to influence the lives of people with mental illness. These stakeholder roles were: educator or staff at an educational institution, employer or human resources staff, health care provider or staff, mental health services provider or staff, employee in some other health or mental health profession, justice system/corrections/law enforcement, lawyer or attorney, journalist or entertainment media, landlord or property manager, policymaker/legislator, or representative of a community or faith-based organization. Participants could select more than one role if appropriate. Each role was a target group for one or more of the SDR program partners conducting trainings. To assess personal and family experience with mental health services, respondents also reported whether they or a family member were consumers of mental health services.

Sample Characteristics

A total of 187 respondents completed an online survey after receiving information or services from IBHP between March 2013 and December 2014. This represents about 14 percent of the individuals who were issued an invitation to complete the survey.

Among respondents, 71 percent reported serving in one of the stakeholder roles that were targeted by IBHP's integrated care effort: being mental health service providers or staff, health care providers or staff, or working in another health or mental health profession. Thirty-five percent of the sample reported being mental health service providers or staff and not serving in any other stakeholder role. A smaller percentage, 7 percent, reported being a health care provider or staff and not serving in any other role. Thirty-seven percent of respondents reported being in some other health or mental health profession or selected multiple health- or mental health—related roles. Thirteen percent reported holding stakeholder roles in fields unrelated to health or mental health. All participants reported holding at least one stakeholder role from the full list provided.

Demographic characteristics of the sample are reported in Table 1. The sample was largely female, between the ages of 26 and 59 years old, and white. Forty-three percent indicated that they or a family member were a consumer of mental health services.

Characteristic	Percentage of Sample
Gender
Female	73
Age
18–25	0
26–59	71
60 and older	29
Race/Ethnicity
White (non-Latino)	66
Latino	9
Black or African-American	4
Asian-American	5
Other*	16

*Respondents who reported being another race, multiracial, or who did not respond.

Results

Information/Services Received

Respondents reported receiving a variety of information or services from IBHP. About one-half (51 percent) participated in an online event, training, or educational presentation, and 32 percent reported participating in an in-person event, training, or educational presentation. Forty-two percent had visited the IBHP website, and 37 percent had received materials or information from IBHP through online, in-person, postal mail, or email channels. Nine percent reported receiving one-on-one technical assistance. However, 11 percent of respondents did not recall receiving any information or services or attending any events; these respondents were not asked additional survey questions.

Variation in the information and services that respondents in different stakeholder roles received are reported in Appendix B.

How Information/Services Were Used

Nearly all respondents reported using the information or services they received in at least one way; only 9 percent reported that they did not use the information. Nearly all respondents—95 percent—also reported being somewhat or very likely to use the information or resources they obtained from IBHP in the future. In addition to the specific uses we described in the survey (see remainder of this section), 16 percent of respondents said that they used the information or services they received “in another way.”

The primary reported use of resources and services was sharing the information with others. Respondents indicated that one of the main ways they used the information or services provided by IBHP was to share it with others (see Figure 1). Sixty-six percent of respondents reported sharing the information and resources that they received from IBHP with colleagues at their own organizations, and 35 percent said that they shared the information with another organization; 16 percent shared information with people with mental illness or family members of people with mental illness.

A substantial minority of respondents reported positive changes in their own behavior as a result of receiving resources and services. In addition to sharing information, respondents reported positive changes in behavior after exposure to IBHP information and services (see Figure 2). Nearly one-quarter of respondents (24 percent) reported acting in ways that are more supportive of people with mental illness. Eighteen percent reported behaving in a way that ensured that someone with mental illness was not discriminated against.

Many respondents reported making organizational or policy changes after receiving information and services. Many respondents reported changes to organizational practice or policies after exposure to IBHP information and services (see Figure 3). Twenty percent reported new collaborations with other organizations, and 18 percent reported making an actual policy or practice change.

Use of information and services varied by stakeholder group. Different stakeholders appear to have used the information and services they received from IBHP in somewhat different ways (see Table 2). However, because our sample is small and possibly not representative of the broader set of recipients of IBHP services and activities, it is unclear whether these results generalize. Health care providers or staff and mental health service providers or staff were less likely to share information with other organizations relative to respondents serving in other health or mental health professions or who did not report working in the fields of health or mental health. A greater percentage of health care providers or staff and mental health service providers reported engaging in supportive behavior as a result of receiving information and services from IBHP, relative to respondents working in other health or mental health roles or outside of the health and mental health fields. A higher percentage of health care providers and staff reported starting new collaborative relationships with other organizations or making policy or practice changes relative to the other stakeholder groups.

	Health Care Providers and Staff* (n = 13)	Mental Health Service Providers and Staff^ (n = 66)	Other Health and Mental Health Professions** (n = 69)	Non-Health or Mental Health Role^^ (n = 25)
Information sharing
I shared the information with colleagues at my organization.	69%	67%	68%	60%
I shared the information with another organization.	23%	26%	41%	52%
I shared the information with people with mental illness or their family members.	31%	11%	16%	8%
Supportive behavior
I acted in ways that are more supportive of people with mental illness.	31%	32%	17%	16%
I behaved in a way that ensured that someone with mental illness was not discriminated against.	31%	24%	12%	16%
Organizational and policy changes
I/My organization started a new collaborative relationship with another organization.	38%	12%	22%	28%
I made an actual policy or practice change.	38%	20%	17%	8%
Other
I used the information in another way.	15%	14%	19%	20%
Did not use the information	8%	5%	12%	12%

* Respondents who endorsed serving in a health care provider or staff role only.

^ Respondents who endorsed serving in a mental health service provider or staff role only.

** Respondents who served in another health or mental health profession, or who endorsed multiple health or mental health roles.

^^ Respondents who served in stakeholder roles in fields other than health or mental health.

Respondents from all stakeholder groups reported being likely to use the information or services received in the future. All respondents who were health care providers or staff or who did not work in health or mental health reported being somewhat or very likely to use the information. Ninety-seven percent of mental health service providers and staff and 92 percent of respondents in other health and mental health professions were somewhat or very likely to use the information in the future.

Cultural Appropriateness and Competence of IBHP's Interventions

Most respondents found the provided information and services to be culturally appropriate. Almost all respondents (98 percent) reported that they would recommend the training, presentation, or information they received to someone of their cultural background (e.g., race, religion, language, sexual orientation, etc.), indicating that respondents felt that the information was culturally appropriate (see Figure 4). In addition, the majority of participants (68 percent) agreed or strongly agreed that the information or resources provided by IBHP were sensitive to their cultural background (race, religion, language, sexual orientation, etc.). Latino and non-White respondents (i.e., participants who reported being Latino; Black or African-American; Asian-American; Native Hawaiian or other Pacific Islander; American Indian, Native American, or Alaska Native; or who described themselves as being of another racial or ethnic group) and non-Latino White respondents endorsed these items at similar levels.

A majority of respondents reported learning more about cultural sensitivity and some reported using more culturally sensitive approaches. Sixty-five percent of participants agreed or strongly agreed that the information or resources provided by IBHP taught them how to be culturally sensitive when talking with someone with a mental illness (see Figure 5). Additionally, 15 percent of respondents reported that exposure to the information and services led them or their organizations to adopt more culturally competent approaches to reducing stigma and discrimination against people with mental illness.

Discussion

We conducted an online survey of 187 recipients of information and services from IBHP to ask how the information and services were used. Though the sample size was lower than desired and may not represent the broader population IBHP served, results of this evaluation suggest that IBHP successfully reached its target audience of professionals in the health care and mental health services fields, and that participants in IBHP's CalMHSA-sponsored SDR efforts made use of the information and services they received from IBHP in ways congruous with the goals of the effort to promote integrated care. For example, approximately 20 percent of respondents indicated that they made policy or practice changes. A similar percentage reported starting a new collaborative relationship with another organization. Though we cannot identify the types of other organizations with which collaborations were initiated, this finding may be indicative of the cross-agency collaboration necessary to provide integrated care. Respondents also reported sharing the information they received within their own organization, with other organizations, or with people with mental illnesses or their family members. This information sharing suggests the potential for greater reach of IBHP's efforts beyond the people who directly received information or services.

This evaluation is subject to several limitations. The survey relied on respondent self-reports of uses of provided information or services, and we have no way to ensure that the reported changes actually occurred. Given that we only surveyed a subsample of recipients of IBHP information and services, we are unsure of the degree to which these results are generalizable to the broader set of recipients of IBHP information and services. Due to the small sample size of the study, we are also unable to draw strong conclusions about variation in the use of information or services among different participant groups (e.g., stakeholders, racial/ethnic groups). Given the broader movement toward integrated care in California (e.g., the Coordinated Care Initiative, implemented in 2012 [California Department of Health Care Services, undated]), it is possible that respondents were exposed to multiple sources of information on implementing integrated care models; we have no way of determining what other integrated care information and services respondents may have been exposed to and the degree to which that may have contributed to reported changes in personal or organizational practices.

Increasing integrated care may improve access to care and health outcomes for individuals with mental and physical health problems. Future evaluation efforts could explore the degree to which professionals in organizations who received information and services from IBHP actually engaged in integrated care and sustained it over time.

The changing of institutional practices and policies surrounding mental illness is a key strategy of CalMHSA's SDR initiative and is designed to complement efforts to reduce stigma-related attitudes, beliefs, and behaviors at the individual and societal levels. CalMHSA's theory of change is that changes in each of these areas will support and increase change in the others, resulting in greater and more sustainable shifts in mental illness stigma across the state. However, it is particularly difficult to determine the effectiveness of policy changes, including integrated care, in reducing stigma over the short term (Burnam et al., 2014; Clark et al., 2013). Future work could examine longer-term consequences of CalMHSA's work through IBHP, as well as consequences of integrated care more broadly, for mental illness stigma. Such additional evaluation efforts could help determine whether integrated care is a valid and cost-effective approach to stigma reduction.

Appendix A

This appendix contains the survey question and response options administered to respondents when asked how they used information or services that they obtained from IBHP. They could check all options that were applicable. Because the follow-up survey was designed for use across multiple SDR program partners, some response options are not directly related to IBHP's integrated care efforts.

Question: How did you use the information or services that you obtained from Integrated Behavioral Health Project in your role or job?

Response options:

• I acted in ways that are more supportive of people with mental illness.
• I behaved in a way that ensured that someone with mental illness was not discriminated against.
• I introduced new policy or legislation to a legislative body.
• I made an actual policy or practice change.
• I shared the information with colleagues at my organization.
• I shared the information with another organization.
• I shared the information with people with mental illness or their family members.
• I/My organization adopted more culturally competent approaches to reducing stigma and discrimination against people with mental illness.
• I/My organization started a new collaborative relationship with another organization.
• I/My organization implemented a stigma/discrimination reduction program.
• I/My organization evaluated a stigma/discrimination reduction program.
• I used the information in another way.
• CHECK HERE IF YOU HAVE NOT USED THE INFORMATION

Appendix B

	Health Care Providers and Staff* (n = 13)	Mental Health Service Providers and Staff^ (n = 66)	Other Health and Mental Health Professions** (n = 69)	Non-Health or Mental Health Role^^ (n = 25)
Attended an event, training, or educational presentation (in-person)	31%	38%	33%	20%
Attended an online event, training, or educational presentation	69%	44%	58%	48%
Received one-on-one technical assistance	0%	14%	7%	4%
Received materials or information (online, in person, or by mail/email)	38%	32%	41%	44%
Visited the website	54%	38%	43%	40%
None of the above	0%	9%	9%	20%

* Respondents who endorsed serving in a health care provider or staff role only.

^ Respondents who endorsed serving in a mental health service provider or staff role only.

** Respondents who served in another health or mental health profession, or who endorsed multiple health or mental health roles.

^^ Respondents who served in stakeholder roles in fields other than health or mental health.

References

Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, Evaluation of the California Mental Health Services Authority's Prevention and Early Intervention Initiatives: Executive Summary and Commentary, Santa Monica, Calif.: RAND Corporation, RR-438/1-CMHSA, 2014. As of October 9, 2015:
http://www.rand.org/pubs/research_reports/RR438z1.html

Butler, Mary, Robert L. Kane, Donna McAlpine, Roger G. Kathol, Steven S. Fu, Hildi Hagedorn, and Timothy J. Wilt, Integration of Mental Health/Substance Abuse and Primary Care, Rockville, Md.: Agency for Healthcare Research and Quality, Evidence Report/Technology Assessment No. 173, 2008. As of October 9, 2015:
http://www.ahrq.gov/sites/default/files/wysiwyg/research/findings/evidence-based-reports/mhsapc-evidence-report.pdf

California Department of Health Care Services, “Coordinated Care Initiative Overview,” web page, undated. As of September 3, 2013:
http://www.dhcs.ca.gov/provgovpart/Pages/CoordinatedCareIntiatiave.aspx

Clark, Wayne, Stephanie N. Welch, Sandra H. Berry, Ann M. Collentine, Rebecca Collins, Dorthy Lebron, and Amy L. Shearer, “California's Historic Effort to Reduce the Stigma of Mental Illness: The Mental Health Services Act,” American Journal of Public Health, Vol. 103, No. 5, May 2013, pp. 786–794.

Colton, C. W., and R. W. Manderscheid, “Congruences in Increased Mortality Rates, Years of Potential Life Lost, and Causes of Death Among Public Mental Health Clients in Eight States,” Preventing Chronic Disease, Vol. 3, No. 2, 2006, p. A42.

Druss, B. G., L. Zhao, S. Von Esenwein, E. H. Morrato, and S. C. Marcus, “Understanding Excess Mortality in Persons with Mental Illness: 17-Year Follow Up of a Nationally Representative US Survey,” Medical Care, Vol. 49, No. 6, 2011, pp. 599–604.

“Farley Health Policy Center,” YouTube Channel, undated. As of October 19, 2015:
https://www.youtube.com/user/CUDFMPolicyChannel

Institute of Medicine, Improving the Quality of Health Care for Mental and Substance-Use Conditions, Washington, D.C.: National Academies Press, 2006.

IBHP—See Integrated Behavioral Health Project.

Integrated Behavioral Health Project, Partners in Health: Mental Health, Primary Care, and Substance Use Interagency Collaboration Tool Kit, 2nd ed., 2013. As of September 3, 2015:
http://ibhp.org/uploads/file/IBHPIinteragency%20Collaboration%20Tool%20Kit%202013%20.pdf

Koyanagi, C., Get It Together: How to Integrate Physical and Mental Health Care for People With Serious Mental Disorders, Washington, D.C.: Barzelon Center, June 2004.

President's New Freedom Commission on Mental Health, Achieving the Promise: Transforming Mental Health Care in America. Rockville, Md.: Department of Health and Human Services, July 2003.

Shim, R., and G. Rust, “Primary Care, Behavioral Health, and Public Health: Partners in Reducing Mental Health Stigma,” American Journal of Public Health, Vol. 103, No. 5, 2013.

U.S. Department of Health and Human Services, Mental Health: A Report of the Surgeon General, Rockville, Md., 1999.

Unützer, J., M. Schoenbaum, B. G. Druss, and W. J. Katon, “Transforming Mental Health Care at the Interface with General Medicine: Report for the Presidents Commission,” Psychiatric Services, Vol. 57, No. 1, 2006, pp. 37–47.

World Health Organization and World Organization of Family Doctors, Integrating Mental Health into Primary Care: A Global Perspective, Geneva, Switzerland: WHO Press, 2008. As of October 9, 2015:
http://www.who.int/mental_health/resources/mentalhealth_PHC_2008.pdf

Notes

[1] The SDR initiative is one of three initiatives included as part of California's statewide prevention and early intervention activities funded by CalMHSA under Proposition 63. The other two initiatives focus on suicide prevention and student mental health.

[2] These represent the subset of CalMHSA-funded IBHP activities for which data were collected and reported in this document. Other IBHP activities that were funded under the CalMHSA SDR initiative, such as regional summits, learning collaboratives, and technical assistance activities, are not addressed here.

[3] A third annual summit was held in 2015, but it fell outside the evaluation time frame and thus is not addressed in this study.

[4] During the evaluation period, this YouTube channel was labeled as the channel for the University of Colorado School of Medicine Department of Family Medicine.

This research was conducted in RAND Health, a division of the RAND Corporation.