Evaluation of the Connections to Care (C2C) Initiative: Interim Report

In any given year, at least one in five adults in New York City is likely to meet the criteria for a mental health diagnosis, yet most do not receive mental health services to treat these problems (City of New York, Office of the Mayor, 2015). Problems such as depression and anxiety disproportionately affect historically underserved segments of the population, such as racial/ethnic minority and low-income individuals, that are most likely to have an unmet need for mental health services (Kataoka, Zhang, and Wells, 2002). To help close these gaps of unmet need, the Connections to Care (C2C) Collaborative—the Mayor's Fund to Advance New York City (Mayor's Fund), the Mayor's Office for Economic Opportunity, and the New York City Department of Health and Mental Hygiene—developed the C2C program. C2C is an innovative model that uses task shifting, an approach to extending evidence-informed health care skills to community-based partners, to integrate mental health support into the work of community-based organizations (CBOs) that do not focus on clinical health care. As one of New York City's ThriveNYC initiatives,¹ C2C aims to remove barriers to mental health care and, ultimately, build community resilience through responsiveness to individual and community needs.

The C2C program recognizes that mental health treatment and behavioral health interventions can involve a wide spectrum of activities to improve individual functioning and promote well-being. Such activities do not necessarily have to involve clinical treatment or occur in a clinical setting. In fact, a range of barriers, including lack of transportation, mental health stigma, and a lack of linguistic or cultural competency among clinicians, may deter some individuals from seeking care in clinical settings, where the majority of mental health services are delivered. Given additional barriers, including a shortage of trained mental health professionals and the costs of treatment that are prohibitive for many low-income populations, it is critical to build capacity in the mental health treatment system in a way that addresses these issues.

Increasing access to mental health supports in nonclinical settings through task shifting is an approach that may directly reduce barriers and expand access to mental health care services. In addition to decreasing logistical barriers such as transportation (clients already work with the nonclinical CBOs), this approach may also decrease the stigma people associate with seeking clinical treatment and help identify groups with mental health needs that may otherwise be missed by the mental health system. By making mental health supports available outside the clinical environment, task shifting may promote earlier needs identification and improve timely access to services that are appropriately matched to needs. In turn, this approach may reduce the adverse effects of treatable behavioral health conditions, enabling individuals to more quickly improve aspects of their quality of life, including their employment and education, and improve overall community well-being.

To build an intervention that operates through nonclinical settings, C2C integrates delivery of mental health skills into social services at 15 CBOs located throughout the city. C2C leverages the existing trusted relationships that New Yorkers already have with participating CBOs. The participating CBOs already provide a variety of services, such as career development, youth services, homeless and domestic violence services, adult education, early childhood services, and services for immigrants. Each CBO contracts with a mental health provider (MHP), developing a close, working relationship essential to ensuring the effectiveness of C2C. The MHP advises CBO staff on effective, evidence-informed interventions, while the CBO tailors the implementation of those interventions to its own context.

In C2C, CBO staff receive training, ongoing coaching, and resources to implement four initial core C2C mental health skills (hereafter, “C2C skills”): mental health screening, mental health first aid (MHFA), motivational interviewing (MI), and psychoeducation (see Table 1 for more information). As their programs mature, CBOs may add other skills beyond these four core skills.

Staff at CBOs participating in the C2C program receive training, coaching, and access to resources to help them deliver, at minimum, four core mental health skills.

1. During mental health screening, CBO staff screen C2C clients for conditions such as depression, anxiety, and substance use and misuse. Staff work with clients who screen positive to explain the results and identify next steps.
2. MHFA teaches non–mental health specialists skills to recognize and respond to signs of mental illness and substance use and misuse. C2C uses MHFA to reduce stigma, promote helping behaviors, and improve knowledge and attitudes about mental health conditions among both CBO staff and clients.
3. MI is a collaborative, person-centered counseling style that aims to help people make and sustain positive behavior changes. C2C uses MI to help clients change or enhance behaviors to improve their lives.
4. Non–mental health specialists can use psychoeducation to provide information to individuals with a mental health condition (and their families) to help them understand the condition and build skills to deal with it optimally. C2C uses psychoeducation to help improve mental health literacy, reduce discomfort about seeking help, and promote healthy behaviors.

As programs mature, the C2C Collaborative encourages CBOs to build on this initial set of skills by adding other appropriate context-driven skills or services to best meet the needs of staff and clients.

CBOs also develop and strengthen pathways to clinical care (e.g., through their MHP partners) that are appropriately matched to individual client needs. C2C, during its five years of operation, expects to serve up to 40,000 low-income New Yorkers (based on goals provided by participating CBOs) among its target populations: (1) young adults between the ages of 16 and 24 who are not attending school and are not employed; (2) unemployed or underemployed adults age 18 or older; and (3) parents/caregivers who are expecting or who have children four years of age or younger.

Each CBO is required to meet certain implementation requirements, such as establishing a formal relationship with an MHP, training staff members in the four core C2C mental health skills, and providing ongoing support and supervision to CBO staff to deliver core C2C skills to clients. By design, the program allots CBOs a great deal of flexibility in site-specific implementation. A CBO has the latitude to navigate its individual relationship with an MHP, set up culturally relevant training and coaching, ramp up staff readiness, and deliver the C2C supports to clients in a way that makes sense for its own organization.

The C2C program calls on both CBOs and MHPs to assume new roles that capitalize on their strengths and capabilities to increase capacity for and reduce barriers to providing mental health services. Their roles were designed so that each would learn from the other: For instance, CBOs could benefit from the mental health expertise of MHPs; in return, MHPs could learn from CBO expertise addressing social determinants of health. CBOs and MHPs become not only service providers, but also teachers who share their expertise. CBO staff leverage trusted relationships with community members and knowledge of community needs to deliver a range of mental health care supports as part of the everyday work they already do, making care more accessible. By integrating the four core C2C skills into their existing services, CBOs may gain more accurate knowledge about mental health and also expand capacity help promote mental health and well-being. C2C equips CBOs to deliver culturally responsive, evidence-informed behavioral health interventions that promote mental health and well-being and identify individuals who need more specialized care and connect them to it. MHPs support CBOs' new capabilities and skills by consulting on implementation design; performing training, ongoing coaching, and monitoring activities; applying clinical judgment to appropriately match mental health services to needs; and providing clinical care when necessary.

Working together, CBO-MHP partners identify clients' needs, select and adapt interventions, and share in both learning and quality improvement. Staff from both types of organizations may also be challenged through training and practice to confront or evolve their attitudes toward mental health issues (e.g., mental health stigma), the way mental health services are typically delivered, and who delivers them. The hope is that C2C's efforts can begin influencing community well-being more broadly, first by the diffusion of knowledge about mental health by trained CBO lay staff, who often live in the communities they serve, and second through the potential downstream effects of CBO clients who have benefited from C2C, as better supporters of their families and neighbors, or even a resource for where to get mental health support.

During the first two program years of C2C, the RAND Corporation has been working with CBOs and MHPs to provide technical assistance (with the McSilver Institute for Poverty Policy and Research at the New York University Silver School of Social Work [NYU McSilver]) and monitor the progress of the program. RAND researchers have also been collecting qualitative and quantitative data to formally evaluate the implementation, impact, and cost of C2C. This interim study describes the methods and plans for these three evaluations, provides initial findings from the implementation evaluation, and presents preliminary descriptive data from the impact evaluation. The data and analysis here derive from years 1 and 2 of C2C, though the specific time frames of data used in this report vary slightly for each evaluation, as described below.

Understanding C2C Implementation

Our implementation evaluation uses data from interviews with key leadership and staff, staff surveys, and quarterly CBO reports to describe how C2C has been implemented within and across the 15 CBOs. The implementation evaluation seeks to understand

• how the C2C program strategies were implemented (including which components were implemented and with what degree of fidelity)
• how MHPs trained and supported CBOs over time
• whether CBO staff knowledge of mental health and C2C skills, as well as attitudes and behaviors about mental health issues and services, showed improvement
• the extent to which CBOs identified clients with mental health or substance use issues as a result of C2C implementation
• the key facilitators of and barriers to effective implementation of C2C program strategies within and across CBO and MHP partnerships.

The preliminary implementation findings in this interim report cover the first set of interviews and surveys—both from summer 2017—and use CBO quarterly report data from March 2016 to March 2018.

Key Implementation Evaluation Findings

Table 2 summarizes the implementation evaluation key findings.

• CBO-MHP Relationships

  • Partnerships encountered initial cultural differences and communications challenges, but developed working relationships over time.
  • Activities to build familiarity within partnerships, such as MHPs sustaining a presence on-site at CBOs, paid dividends in bridging divides.

• Training

  • More than 1,200 staff and supervisors were trained in at least one core C2C skill, which surpassed targets for the end of year 2.
  • MHPs delivered the bulk of training, especially in year 1.
  • About one year into implementation, more than two-thirds of staff were satisfied with the trainings they had received.

• Coaching and Supervision

  • In the 2017 staff survey, about half of staff reported receiving coaching or supervision only once or twice; a small number (6 percent) received it more than ten times.
  • The C2C Collaborative updated its guidance to CBOs on these activities in May 2017 to help CBOs and MHPs develop robust ongoing coaching and supervision structures.
  • Most of the 2,110 coaching hours reported between March 2016 and March 2018 were logged in year 2 of implementation.

• Staff Readiness

  • About half of staff surveys indicated high confidence in using a given mental health skill with a client.
  • One year into implementation, trained individuals reported feeling more comfortable delivering MHFA and MI than screening or psychoeducation.
  • Approximately three-quarters of staff respondents felt they had the right resources and support to deliver C2C skills and refer clients to intensive treatment when needed.

• C2C Delivery

  • Core C2C skills were delivered to 16,701 unique clients in the first two years, which is 98 percent of the new client target for that stage of the program.
  • By the end of year 2, staff members had delivered services to an average of 70 percent of clients across CBOs.
  • As of March 2018, CBOs referred 2,254 clients for mental health treatment; 60 percent of referrals resulted in at least one visit to a mental health provider (the C2C Collaborative set a goal of 70 percent of referrals resulting in at least one visit).

• Early Perceptions of Impact

  • Interviews indicated that staff and CBO leadership perceived a positive effect of C2C on mental health issues for clients and themselves.
  • Client and staff interviewees noted changes in the way they perceive, discuss, and approach mental health issues.

CBO-MHP Relationships

One of the first implementation decisions made by all CBOs was the selection of an MHP. Some CBOs chose to partner with an organization with which they had a prior informal relationship (e.g., as a referral source). A few CBOs partnered with a provider under the same umbrella organization but with which a formal relationship did not exist; others forged a relationship with an organization with which they had never worked before. Specific implementation arrangements between CBOs and MHPs vary across organizations, including the types of CBO and MHP staff members who deliver supports, when and how supports are delivered, and the roles of each CBO and MHP with respect to training and coaching/supervision activities.

A number of sites did report some anticipated early challenges with navigating the CBO-MHP roles, including organizational culture differences and difficulties with communication between organizations. For example, MHPs were often charged with selecting and delivering training, but many initially lacked the context needed to tailor these trainings to CBO environments. In addition, while MHPs supervised CBO staff for implementation of C2C skills, they did not have the authority to require staff to attend trainings or individual coaching sessions.

CBOs and MHPs implemented several strategies to overcome these challenges, including inviting MHP staff to spend time on-site at the CBO to understand client flow and staff roles, establishing open lines of email and phone communication, scheduling structured meetings to discuss implementation and assign tasks, embedding MHP staff on-site at CBOs, specific trainings for CBO staff in the MHP intake processes, and soliciting staff feedback on training to tailor future materials. Most interviewees noted that learning about each other's organizations and keeping the lines of communication open were time-consuming but helped in building a strong relationship and a program that met the needs of clients and staff.

Training

Most training in the four core C2C skills took place either internally at the CBOs or together with MHPs, with some CBOs relying more heavily than others on MHPs to deliver training. In addition, MHFA and MI trainings were offered through external organizations, sponsored by ThriveNYC and the Mayor's Fund, respectively, and most CBOs utilized these supplemental training resources. By the end of year 2, more than 1,200 staff and supervisors had received training in at least one support, with most of that training occurring in year 1. The overall number of staff trained exceeded year 2 targets. In that same time frame, almost 250 staff and supervisors had been trained in all four core C2C skills. More than two-thirds of staff reported satisfaction with the training they received and a desire for more training. Satisfaction was associated with MHP and CBO collaboration to tailor training to a CBO's cultural context and participant population and with CBO responsiveness to staff feedback about training needs. Turnover in staff among some CBOs—a problem that preceded the start of C2C—remains a significant challenge to maintaining a sufficient number of trained staff in a handful of organizations and makes the training cycle seem continuous for some CBO leaders.

Coaching and Supervision

CBOs and MHPs developed plans for follow-up support—coaching to reinforce the four core C2C skills and supervision of trained staff—that varied according to organizational needs, site-specific implementation plans, and staff needs, availability, and capacity. Sites could engage MHP and/or CBO staff who had undergone supervisory training to provide coaching and supervision in the four core C2C skills. In the summer 2017 CBO staff survey, about half of the respondents reported receiving coaching or supervision only once or twice, and almost 20 percent reported receiving none. Conversely, a small proportion (6 percent) received coaching or supervision more than ten times. Nearly 80 percent of the 2,110 coaching hours logged by March 2018 occurred in year 2, after the C2C Collaborative issued a clarification of follow-up support requirements (coaching/supervision to be delivered on a quarterly basis, at minimum) and after CBOs and MHPs had sufficient time to develop coaching processes. MHPs delivered most of this coaching, though the proportion of CBO-administered coaching hours increased substantially by the end of year 2. Data from the next staff survey, which is currently under way, will clarify whether CBO staff continued to increase the frequency of providing coaching and supervision. As with staff perceptions of training, most CBO staff indicated both satisfaction with the supervision received and a desire for additional coaching.

Staff Readiness to Deliver C2C Skills

Among 2017 CBO staff survey respondents, about half of CBO staff trained in a given C2C skill reported high confidence in using that skill with clients. This suggests that, while many staff felt ready to use their new C2C skills in practice, a substantial proportion did not yet feel confident using them. Among individuals who had received training, respondents reported greater comfort with administering MHFA, MI, and screening than with psychoeducation; changes in C2C Collaborative-provided guidance regarding psychoeducation were issued in year 2, and most sites were in the process of responding to that guidance at the time of the survey. In interviews, some CBO staff reported feeling uncomfortable talking with their clients about mental health issues early in the implementation process, but noted that experience using C2C skills with clients and ongoing coaching and supervision helped ease that discomfort. Indeed, the majority of staff survey respondents (ranging from 70 to 85 percent) endorsed survey items regarding feeling comfortable talking about mental health issues with clients and feeling supported by their organization, as well as about having access to resources needed to help their clients and assist with referrals to more-intensive mental health treatment.

C2C Delivery

Over the first two years of the program, staff at the 15 CBOs delivered the four core C2C skills to 16,701 unique clients. The proportion of CBO clients receiving services from a CBO staff member who was trained in C2C skills rose substantially during this period, from 22 percent in year 1 to 70 percent in year 2. Many CBOs began delivering MHFA and screening in year 1, with depression as the most common mental health condition for which clients were screened. As of summer 2017, CBO direct service staff reported spending an average of more than eight hours per week delivering one or more C2C skills to clients and assisting with referrals to mental health services at the MHP or an external mental health provider. In year 2, most CBOs were implementing all four core C2C skills and began to enact more robust coaching and supervision programs to support staff in the delivery of C2C skills to clients. As of March 2018, more than 2,000 clients had been referred for mental health treatment, and almost 60 percent of those referrals were completed with at least one visit with a mental health provider. This is lower than the target set by the C2C Collaborative at the start of the project, which is to see 70 percent of referrals result in at least one visit. Although some MHPs adopted more flexible policies to accommodate C2C clients, and CBOs tried to minimize barriers to accessing more-intensive mental health treatment, some challenges remained, including client stigma; privacy concerns; and a lack of resources such as transportation, money or insurance, and child care. Future data will shed light on whether CBO-enacted strategies targeting these barriers are successful.

Perceptions of C2C Early Impact

Although we have not yet conducted a detailed assessment of client outcomes (see impact evaluation below), key informant interviews conducted thus far showed early evidence of the perception that the C2C program is having a positive effect on mental health issues (CBO [ n = 35] and MHP [ n = 29] leaders, CBO staff [ n = 80], and clients [ n = 38]). The interviewees also indicated that C2C is creating a cultural shift in their places of work, giving them a common language with which to discuss behavioral health issues, as well as an understanding of how to approach their clients in a different way. Both CBO leaders and staff noted that trainings and C2C skill delivery also enhanced their own mental health and stress management. CBO leaders mentioned C2C's facilitation of shared resources and best practices among a wider network of like-minded organizations. Interviewees cited instances of clients benefiting from C2C skills, especially in terms of overall well-being, access to mental health resources, and stigma reduction. In addition, clients themselves shared examples of C2C's positive effects on their lives, including improved parent-child and other interpersonal relationships, improved understanding of their mental health, greater confidence, and greater access to long-term care.

Key Impact Evaluation Findings

Table 3 summarizes the impact evaluation key findings.

• Survey Enrollment

  • Eligibility screenings have been completed by 1,234 participants from C2C CBOs (805 clients) and comparison CBOs (429 clients).
  • Baseline surveys have been completed by 86 percent of those eligible (634 from C2C CBOs and 267 from comparison CBOs), above the target of 80 percent of those eligible.
  • Follow-up surveys will be conducted six and 12 months after baseline surveys.

• Demographics (among C2C clients only)

  • Most participants are over 18 years of age and from ethnic minority backgrounds.
  • Over half report earning less than $5,000 per year.
  • Baseline rates of moderate to severe mental health issues (depression, anxiety, alcohol/substance abuse, PTSD) ranged from 22 to 30 percent; over half report reluctance to seek mental health care.
  • In the past six months, more than one-quarter had received inpatient mental health care or had seen a mental health provider.

The impact evaluation uses a client survey and other data to analyze the experiences and outcomes of participants at 13 of the 15 CBOs engaged in the C2C study and compares them with participants at ten CBOs offering similar social services but without the mental health support integration. Potential client survey participants at C2C and comparison CBO sites are screened for eligibility (i.e., fluent in English or Spanish, presence of mental health symptoms) and are enrolled if eligible and willing. Enrolled participants complete a baseline survey and follow-up surveys six and 12 months after baseline. The baseline and follow-up surveys are being completed on a rolling basis, and most participants have not yet completed follow-up surveys. The impact evaluation aims to enroll approximately 2,000 participants in total (1,000 in each study arm) in the client survey, which is a subset of the population receiving C2C services overall.

Although the data used at this interim stage for the impact evaluation are too preliminary to draw any conclusions about the sample overall, we are able to report some descriptive characteristics of the study participants who enrolled in the evaluation between June 2017 and July 2018. Thus far, 1,234 clients have been screened from the C2C CBOs (805 clients) and the comparison CBOs (429 clients). Of those eligible, 86 percent enrolled in the study and completed baseline surveys (n = 634 C2C and n = 267 comparison participants), which exceeded the target of 80 percent of those eligible. The interim report analyses focused exclusively on the 634 clients of C2C CBOs who completed the baseline survey (i.e., not the comparison CBO participants). Most participants from the C2C CBOs were over 18 years of age and from ethnic minority backgrounds, and over half reported incomes of less than $5,000, which indicates that C2C appears to be reaching its intended target population. Baseline rates of moderate to severe mental health issues (based on previously established scoring criteria) included 22 percent for depression, 30 percent for anxiety, 20 to 30 percent for alcohol and substance abuse, and 30 percent for post-traumatic stress disorder (PTSD). Many participants expressed ambivalence about seeking mental health care in the baseline survey, and well over half endorsed statements regarding keeping feelings, emotions, or thoughts to themselves and wanting to solve problems on their own. More than one-quarter of the sample had received inpatient care for a behavioral health problem, and about the same proportion had seen a mental health provider in the past six months.

Study enrollment is expected to continue through March 2019. Until that time, more baseline surveys will be completed, and the initial study participants will begin to complete follow-up surveys. The data from these subsequent surveys will provide more information on the impact of the C2C program.

Cost Evaluation

An evaluation of the costs—and possibly overall savings—associated with C2C is also underway. The cost evaluation uses data from multiple sources to estimate the startup and maintenance costs of running the C2C program, as well as the program's effects on the cost of other social services due to improved mental health outcomes. We will collect data from financial reports submitted by CBOs to the C2C Collaborative, annual nonlabor expense and compensation reports, biannual cost surveys, assessments of participants in the impact evaluation, and administrative data from government agencies to conduct these analyses. While it is too early at this time to report on this aspect of the study, the cost evaluation will quantify the average cost of C2C per CBO on both a quarterly and yearly basis; the total cost of the C2C program (including cost of labor, payments for contracted services such as payments to MHPs, and direct and indirect costs); and the average cost per client served. RAND will also estimate the net effect of C2C on both medical spending and government expenditures at the city, state, and federal level.

Looking Ahead

At the time of this writing, CBOs and MHPs have worked together to address early implementation challenges and are focusing on full implementation of the C2C program. Over the next two years, they will continue to refine their processes and increasingly consider program sustainability. Simultaneously, we will continue our evaluations, documenting C2C implementation, impact, and cost. Implementation and cost evaluation data collection, including staff surveys, key informant interviews, and CBO program data, will continue through 2019. Impact study participant enrollment at both C2C and comparison CBO sites will continue through March 2019, with follow-up assessments ending in March 2020. A final report will be made available in 2020.