Across the education, public health, and human and social services arenas, there has been renewed interest in bringing agency representatives together to work on the promotion of student mental health and wellness. When effective, it is believed that collaboration among agencies can build cross-system partnerships, improve referral processes and effective use of resources, increase access to services, enhance professional development and the quality of service provision, and ultimately improve child and family outcomes.
One of the aims of California's Statewide Prevention and Early Intervention (PEI) K–12 Student Mental Health (SMH) initiative—funded under Proposition 63—was to build collaboration among K–12 school districts, counties, and regions to increase the effectiveness of SMH programs across the state. The California Department of Education (CDE) established the Student Mental Health Policy Workgroup (SMHPW) to develop statewide policies that would impact K–12 students, with a focus on enhancing linkages to services provided at schools or through the foster care system, county departments of mental health, special education programs, and community-based organizations. In addition, the California County Superintendents Educational Services Association (CCSESA) led California Mental Health Services Authority (CalMHSA)—sponsored efforts to organize and support the County Cross-System Leadership Consortia. Each of California's 58 counties had an identified consortium, led by a county office of education representative (COE Lead) who was responsible for convening meetings and reporting activities to the CCSESA regional lead in each of the 11 CCSESA regions.
As part of the evaluation of activities funded under Proposition 63, researchers from SRI International and the RAND Corporation evaluated the development, quality, and effectiveness of collaboration among both the SMHPW and county consortia. The current report focuses on findings from collaboration surveys; early findings from key informant interviews are available in a separate report (www.rand.org/t/rr688).
The Collaboration Surveys
SRI researchers developed and administered a brief collaboration survey for participants in SMH collaborative activities, including members of the SMHPW and county consortia. The collaboration survey items were derived from validated surveys found in the current literature (e.g., Bartsch et al., 2012; Larson and Hicks, 2010; Thomson, Perry, and Miller, 2009). The survey was administered electronically via email, included approximately 25 items, and took ten to 15 minutes to complete. Versions of the survey differed slightly in terms of wording to customize items for particular respondent groups, but, in general, questions included Likert-rated items on a three-point scale: not at all, to some extent, and to a great extent. Respondents rated three major constructs: (1) the governance, function, and goals of collaborative activities; (2) the extent to which they believed that collaboration activities were associated with improvements in the delivery of SMH PEI services and supports; and (3) their perceptions of ongoing challenges and facilitators to collaboration, their personal level of participation in the group, and their perceptions of the group's achievement of goals.
The SRI research team consulted with the CDE and CCSESA program staff to identify individuals who participated in CalMHSA-supported collaborative activities and represented different program components and roles in various locations. These individuals were invited to participate in the surveys due to their personal and professional involvement in SMH partnerships and were in positions to judge the benefits, challenges, and achievements of collaboration.
Each potential respondent was asked to give consent (electronically) for his or her participation in the survey. No incentives were offered for participation, and individuals could refuse to participate and/or refuse to answer any question in the survey. Survey items asked participants to identify themselves by gender and age categories and to provide brief demographic information (e.g., county location of workplace). There were no unique links between participants' responses and identifying information, however, so responses were not attributable to specific individuals.
Responses from the SMHPW and county consortia members who completed the survey are reported separately below.
Findings from the Student Mental Health Policy Workgroup
Tom Torlakson, the state superintendent of public instruction (SSPI), convened the SMHPW to develop a framework for student mental health, identify best practices, and recommend policies at the state, regional, and district levels. Members met quarterly and represented multiple sectors and consumers of the mental health system, such as state and county mental health professionals, school administrators from K–12 to higher education, school nurses, school psychologists, representatives from California higher education systems, community-based organizations, consumer and advocacy groups, youth, family members, and researchers. The SMHPW developed and maintained strong partnerships with statewide agencies and other organizations, such as the National Alliance on Mental Illness and its local affiliates, the California Commission on Teacher Credentialing, the California State Parent-Teacher Association, the California Association of School Counselors, the California Association of School Social Workers, the California School Boards Association, the California School-Based Health Alliance, the Mental Health Association in California, the California Youth Empowerment Network, and United Advocates for Children and Families.
SMHPW survey respondents. Demographic information regarding the SMHPW survey respondents is provided in Table 1. All 35 members of the SMHPW were invited to participate in the collaboration survey; 18 members responded.
Table 1. SMHPW Survey Respondents
| Description | n | % | ||
|---|---|---|---|---|
| Total Participants | 18 | 100 | ||
| Gender | ||||
| Female | 9 | 50 | ||
| Male | 9 | 50 | ||
| Age | ||||
| 26–35 | 2 | 11 | ||
| 36–45 | 2 | 11 | ||
| 46–59 | 8 | 44 | ||
| 60+ | 6 | 33 | ||
| Race/Ethnicity^ | ||||
| White | 13 | 83 | ||
| Black | 2 | 11 | ||
| Latino | 2 | 11 | ||
| Asian | 1 | 6 | ||
^ Respondents could select more than one response.
More than one-third (39 percent) of the SMHPW respondents had been involved with the CalMHSA project just one to two years; 28 percent were involved two to three years, 17 percent more than three years, and 17 percent were involved for less than one year. Most respondents (83 percent) reported that, in the past six months, they participated regularly in scheduled in-person meetings with their partners, whereas only 17 percent attended scheduled meetings infrequently.
Governance and structure of the SMHPW. More than one-half of the SMHPW members (61 percent) reported that they rely on informal personal relationships with their partners to work together; in contrast, only 27 percent of respondents said they relied on formal agreements, such as memoranda of understanding or contracts. All members (100 percent) believed that they had consistent opportunities to share information with partners about resources and capabilities, and most (83 percent) problem solved together with their fellow members to develop solutions to student mental health issues.
When asked about the partnership process and what brings the collaborative members “to the table” together, all respondents (100 percent) reported that the advantages of collaboration outweigh the disadvantages. Most members (83 percent) also agreed that they had combined and used each other's resources so that they all benefited from the collaboration, and about one-half believed that their fellow members shared similar goals and had clearly defined roles in the SMHPW (56 percent and 44 percent, respectively).
Function of the SMHPW. When asked which area was the focus of their collaborative work, a majority of SMHPW members indicated that identifying and promoting best practices and informing audiences of available services were frequent activities of the group (72 percent and 61 percent, respectively); one-half (50 percent) also indicated that coordinating services and supports as well as developing collaborative learning opportunities for their members were also focus areas.
The nature of SMHPW collaboration. SMHPW respondents agreed that there were multiple facilitators of collaboration (Figure 1), including agency support to pursue interagency work (72 percent), similar populations of interest (72 percent), and ease of information sharing (56 percent).
Figure 1. Facilitators of Collaboration
There were two factors that more than one-third of respondents (39 percent) agreed were serious challenges to collaboration: lack of staff time or resources to support interagency work, and turf issues—such as competing for clients or funding. Most other factors, such as no preexisting relationships or difficulty sharing information, were rated as serious challenges by no more than 11 percent of responding SMHPW members (Figure 2).
Figure 2. Challenges to Collaboration
Impact of the SMHPW. Most (70 percent) of the SMHPW members reported that their collaboration resulted in improved student mental health outcomes. In addition, about one-half (46 percent) of the respondents reported that their fellow members were planning for sustainability of SMH PEI services beyond the period of CalMHSA grant funding.
When asked the degree to which they believed that collaboration resulted in improved quality of services and supports along various dimensions, nearly one-half of the SMHPW respondents reported that they were successful in improving consistency in policies related to SMH (47 percent), coordinating and sustaining services and supports (47 percent), and improving the cultural competence of services for diverse groups (44 percent; Figure 3).
Figure 3. Improved Quality of Services
When asked the degree to which CalMHSA provided resources necessary to pursue collaborative activities that otherwise would not have occurred, 77 percent of SMHPW respondents indicated “a great extent,” suggesting the important role that CalMHSA support has played in forming and maintaining collaborative relationships to improve K–12 student mental health outcomes across California.
Findings from the County Consortia
Through CCSESA, county superintendents have a regional structure and statewide organization to deliver educational support services to all of the state's districts and communities. CCSESA's Regional K–12 SMH Initiative was based on a common statewide framework of SMH PEI strategies that also supported regional flexibility to build on local strategies and to employ culturally competent practices. County consortia, which consisted of representatives from organizations including county mental health, probation, school districts, foster care, and youth agencies, worked together locally and regionally to build cross-system collaboration, education and training, technical assistance to schools, and school-based demonstration projects. County consortia developed their own SMH PEI plans and capacities to meet local SMH priorities and needs while also having the opportunity to connect with other counties to gain insights on lessons learned and model practices that address common issues. Some county consortia existed prior to the CalMHSA initiative; others were formed or expanded to meet local needs and regional goals.
County consortia survey respondents. SRI researchers invited respondents from all regions in the state via a stratified random sample. In each of the 11 CCSESA regions, SRI randomly selected two county consortia to participate (except in the case of Region 11, which consists solely of Los Angeles County). The final sample pool included consortia from 21 counties; and in the end, 113 consortia members responded to the survey. Table 2 provides demographic information for the county consortia survey respondents.
Table 2. County Consortia Survey Respondents
| Description | n | % | ||
|---|---|---|---|---|
| Total Participants | 113 | 100 | ||
| Gender | ||||
| Female | 85 | 75 | ||
| Male | 28 | 25 | ||
| Age | ||||
| 26–35 | 8 | 6 | ||
| 36–45 | 26 | 23 | ||
| 46–59 | 53 | 47 | ||
| 60+ | 26 | 23 | ||
| Race/Ethnicity^ | ||||
| White | 93 | 82 | ||
| Latino | 16 | 14 | ||
| Asian/Pacific Islander | 8 | 7 | ||
| Black | 6 | 5 | ||
| American Indian/Alaska Native | 5 | 4 | ||
| Other | 5 | 4 | ||
^ Respondents could select more than one response.
About one-third (32 percent) of the respondents had been involved with their consortium more than three years; 27 percent were involved one to two years, 23 percent for two to three years, and 19 percent were involved for less than one year. More than one-half of respondents (58 percent) reported that, in the past six months, they participated regularly in scheduled in-person meetings with their partners, 28 percent attended scheduled meetings infrequently, and 14 percent had not met their partners in person.
Governance and structure of the county consortia. More than one-third (36 percent) of the county consortia members reported that they rely on informal personal relationships with their partners to work together, and 37 percent of respondents said they relied on formal agreements, such as memoranda of understanding or contracts. A majority of respondents (57 percent) reported that they had consistent opportunities to share information with partners about resources and capabilities, and 60 percent reported that they problem solved together with their partners to develop solutions to student mental health issues.
When asked about the partnership process and what brings the collaborative organizations “to the table” together, respondents overwhelmingly (86 percent) reported that the advantages of collaboration outweigh the disadvantages. Almost one-half of consortia members (47 percent) also agreed that their partners share similar goals and activities, and that they had combined and used each other's resources so that they all benefited from the collaboration (55 percent); however, only about one-third (35 percent) reported that they had clearly defined roles and responsibilities within their consortium.
Function of the county consortia. When asked about which area was the focus of their collaborative work, about one-half of consortia members indicated that informing K–12 audiences of available services, identifying and promoting best practices, and coordinating services and supports were frequent activities of the workgroup (52 percent, 50 percent, and 46 percent, respectively).
Nature of county consortia collaboration. There were multiple factors that more than three-quarters of consortia respondents agreed were facilitators of collaboration (Figure 4), including similar populations of interest (78 percent), similar or complementary program goals (77 percent), and a history of working together (76 percent). Also, a majority of respondents rated agency support (62 percent), similar policies (62 percent), and ease of information sharing (60 percent) as helpful to building partnerships.
Figure 4. Facilitators of Collaboration
There was one factor that more than one-third of respondents (37 percent) agreed was a serious challenge to collaboration: lack of agency resources (i.e., staff time) to support interagency work (Figure 5). All other factors, such as no preexisting relationships or difficulty sharing information, were rated as serious challenges by less than one-fifth of the county consortia respondents.
Figure 5. Challenges to Collaboration
Impact of the county consortia collaboration. About one-third (34 percent) of the consortia members reported that their collaboration resulted in improved mental health outcomes among students served. In addition, two-fifths (40 percent) of the respondents reported that their partners were planning for sustainability of SMH PEI services beyond the period of grant funding, and 59 percent believed that the collaboration would likely remain strong and effective in efforts to address SMH even after the CalMHSA grant funding ended.
When asked the degree to which they believed that collaboration resulted in improved quality of services and supports along various dimensions, about one-half or more of the consortia members believed that they were successful in coordinating services (56 percent), expanding services (50 percent), and enhancing referrals to SMH services and supports (49 percent). More than one-third of respondents also believed that they promoted more consistency in policies (39 percent) and improved the cultural competence of services for diverse groups (38 percent; Figure 6).
Figure 6. Improved Quality of Services
We examined whether respondents' beliefs about facilitating or challenging factors in the success of their collaboration were associated with their beliefs in the collaboration's impact on SMH outcomes. Results indicated there was a strong relationship between respondents' belief in the importance of similar policy and advocacy goals and their belief that the group's collaborative work ultimately improved SMH outcomes (χ2 = 5.59, p < 0.05). Conversely, consortia members also thought that lack of preexisting relationships and prior challenges in their history of working with other consortia members could thwart their collaboration's impact (χ2 = 4.65 and χ2 = 3.86 respectively, p < 0.05; Table 3).
Table 3. Factors Affecting the Impact of CCSESA Collaborations
| Factor Facilitating or Challenging Collaboration | Collaboration Improved SMH | χ2 |
|---|---|---|
| Similar Policy Goals | 43% | 5.59 (p < 0.05) |
| No Preexisting Relationship | 0% | 4.65 (p < 0.05) |
| Prior Challenges with Partner | 0% | 3.86 (p < 0.05) |
When asked the degree to which CalMHSA provided resources necessary to pursue collaborative activities that otherwise would not have occurred, 60 percent of county consortia respondents attributed “a great extent” of their capacity to the support provided by CalMHSA. This finding again underscores the role that CalMHSA has played in supporting collaborative relationships to improve K–12 student mental health outcomes across California.
Summary
Both county consortia and SMHPW members agreed that there were many factors that facilitated collaboration, such as a history of working together, similar program and policy goals, similar populations, and agency support to pursue interagency work. One serious challenge noted by many consortia members and SMHPW members was the lack of staff time or resources to support collaboration; a significant proportion of SMHPW members also revealed that turf issues could thwart the success of partnerships.
Overwhelmingly, respondents in both groups reported that the advantages of collaboration outweighed the disadvantages. Many respondents attributed improvements in SMH outcomes to collaboration, including improved coordination of available SMH supports and services and consistency in policies related to SMH. County consortia respondents were more likely to attribute improved outcomes to collaboration if they also felt they had similar policy and advocacy goals and a history of working well together. Finally, about one-half of the respondents in both groups reported that they were planning for sustainability and believed their collaborations would remain strong even after the CalMHSA grant funds ended.
References
Bartsch, D., A. Keller, P. Chung, and C. Armijo, Improving Access to Health Through Collaboration: Lessons Learned from The Colorado Trust's Partnerships for Health Initiative Evaluation, Denver, Colo.: The Colorado Trust, 2012.
Larson, C., and D. Hicks, “Collaboration Survey,” in OMNI, House Bill 1451 Collaborative Management Program: Collaborative Effectiveness Surveys—Initial Data Summary 2010, December 14, 2010, pp. 20–25.
Thomson, A. M., J. L. Perry, and T. K. Miller, “Conceptualizing and Measuring Collaboration,” Journal of Public Administration Research and Theory, Vol. 19, No. 1, 2009, pp. 23–56.
This research was conducted in RAND Health, a division of the RAND Corporation.
