Evaluation of Mental Health First Aid in New York City

Mental health challenges, including major depressive disorder and serious mental illness, are prevalent among the population of New York City (NYC), yet nearly half of adults who experience such challenges encounter barriers that can significantly delay access to needed mental health services or never receive them at all. In 2015, the de Blasio administration launched the Mayor's Office of Community Mental Health (OCMH),¹ a landmark effort made up of more than 50 initiatives that was designed to provide comprehensive, inclusive, and holistic mental health programming through innovative partnerships between city government agencies and community-based organizations (CBOs) in the city. One of these initiatives centered on a citywide rollout of training in Mental Health First Aid (MHFA), an eight-hour course intended to equip nonclinical individuals to identify, understand, and respond to the signs and risk factors for mental illness and substance use disorders.

The Department of Health and Mental Hygiene (DOHMH) had been offering MHFA trainings, but this effort grew exponentially under OCMH, which launched the citywide rollout in 2016 with a goal of training 250,000 New Yorkers by the end of 2020. Citywide trainings were facilitated through city agencies and CBOs with the intention of transforming the culture around mental health within agencies and communities in NYC. More than 25 city agency partners were involved in the dissemination of MHFA. During the height of the program, trainings were offered citywide seven days a week to various populations of focus in multiple languages. By early March 2020, more than 155,000 had been trained in MHFA. Trainings continued until the start of the coronavirus disease 2019 (COVID-19) pandemic in March 2020, at which time all trainings were suspended.

During the March 2020 COVID-19 shutdown, planning for an independent evaluation of the MHFA program was underway but had not yet been launched. In November 2019, the Mayor's Office for Economic Opportunity (NYC Opportunity), in collaboration with DOHMH and OCMH, commissioned the RAND Corporation to conduct an evaluation designed to assess the impact of MHFA at the individual trainee, agency, and community levels. To assess the impact of MHFA at the individual trainee level, a quasi-experimental longitudinal survey of individuals who registered for MHFA training was planned. To gauge agency impact, a survey of employees at a set of city agencies to compare high and low doses of MHFA training was also planned. Finally, to assess community-level impact, the evaluation planned to conduct focus groups with a set of CBOs to compare high and low doses of MHFA training. The evaluation plan had to be adapted given the pause of MHFA trainings. When this report was written (early 2022), in-person MHFA training had not resumed, and DOHMH was determining future mental health education programming.

This study describes the results of the adapted evaluation, a mixed-methods study that included a web-based survey of individuals who had received MHFA training since the inception of the program and a series of five focus groups with leaders of CBOs that provide services to underserved community groups in NYC and staff from a single city agency. These two activities collected data from individuals involved with MHFA to assess the program's impact at the individual, city agency, and community levels and to examine the needs and desire for future mental health training offerings.

Web Survey

The survey of trainees was fielded to all individuals who had received MHFA training through the city at any time in the more than four years of offerings. The survey ran from July 27 to August 27, 2021; it took about 15 minutes to complete. Email invitations were sent to a total of 130,020 individuals, of which 17,890 reported being currently employed by one of the 26 city agency partners (referred to as city agency employees), and 112,130 were not employed by a city agency partner (referred to as community-based trainees). Community-based trainees were members from the community who are affiliated with CBOs (26 percent), faith-based organizations (6 percent), other nonspecified organizations (41 percent), and other city agencies (4 percent) and who had no work affiliation (23 percent). Community-based trainees also represented a variety of occupational fields, such as mental health and social services (19 percent), education (17 percent), health care (16 percent), law enforcement/public safety (2 percent), other (26 percent), or no occupation (15 percent). Because city agency employees were prioritized for MHFA training and were a primary group of interest (because of OCMH's focus on changing the culture of government agencies, providers, and CBOs), the survey targeted approximately equal numbers of city agency employees and community-based trainees. We aimed for a total of 1,500 city agency employee respondents and 1,500 community-based trainee respondents to ensure adequate sample sizes to conduct comparisons between city agency employees and community-based trainees and to assess factors associated with MHFA-related outcomes within these two subsamples. The overall survey response rate was approximately 2.1 percent (6.2 percent for city agency employees; 1.4 percent for community-based trainees). Response rates should be interpreted with caution because of limitations of the study design (e.g., the community-based trainees survey was closed after reaching the target number of respondents). Based on available DOHMH administrative data on trainee demographic characteristics, the survey sample was generally representative of the broader population of MHFA trainees, although survey respondents tended to be slightly older and reported higher educational attainment. The final analytic sample included 2,639 trainees (1,084 city agency employees; 1,555 community-based trainees). It assessed training-related outcomes specific to MHFA, including respondents' mental health knowledge, stigmatizing attitudes, their helping behaviors, the reach of MHFA within their social network, and the impact on mental health service use among individuals with whom respondents applied MHFA. It also asked about respondents' well-being, their perceptions of their workplace mental health climate, and their perceptions of MHFA training and future training needs.

This adapted evaluation used the web survey data to assess the impact of MHFA at the individual level (by analyzing the full sample of trainees), at the agency level (by analyzing the subset of city agency employee trainees), and at the community level (by analyzing subgroups of trainees by sociodemographic characteristics such as race/ethnicity and gender identity). Item-level descriptive statistics, logistic regressions, and analyses of variance were conducted to characterize responses and explore differences in outcomes by factors such as training experiences, occupation, agency affiliation, and sociodemographic characteristics.

Focus Groups

The RAND evaluation team, together with NYC Opportunity, OCMH, and DOHMH stakeholders, recruited focus group participants who were either

• leaders of CBOs that support four underserved communities (Latinx, African American, Chinese, and sexual and gender minorities [SGMs, representing the LGBTQ+ community])
• city agency employee frontline staff (participation was only possible from the Human Resources Administration/Department of Social Services, a city agency with high survey response rates).

A total of five focus groups (four CBO groups and one city agency) were held virtually between June and November 2021 and were composed of three to four individuals each. The research team used the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to develop a semistructured interview protocol specific to the MHFA evaluation questions. All discussions were audio recorded and professionally transcribed. The transcript for the focus group conducted in Spanish was also professionally translated.

Analyses of the deidentified transcripts involved data coding to uncover emergent themes from the discussions. Thematic analysis focused on participants' experiences of MHFA implementation or participation, their motivation and outcomes sought for participating in MHFA, trainees' use of MHFA, and their wishes for future trainings or mental health resources.

Key Findings

Findings from this evaluation indicate that the use of city-sponsored MHFA training may be a valuable tool for building support skills in the community. By diffusing MHFA through city agency workplaces and CBOs, tens of thousands of New Yorkers were given tools to come to the aid of individuals in their personal and professional lives who are experiencing mental health challenges. Here we summarize key findings synthesized from survey and focus group data that are organized according to MHFA-related outcomes at the individual, city agency, and community/sociodemographic levels. Individual-level findings are based on the survey data among the full sample of respondents, city agency findings are derived from survey data on the subset of city agency employees and the focus group with a single city agency, and community-level findings are drawn from the survey data among community/sociodemographic subgroups and the four focus groups with leaders of CBOs.

Individual

• Most survey respondents (90 percent) indicated using MHFA skills (active listening, providing reassurance and information, and encouraging professional help-seeking) in the last six months. About 77 percent of respondents felt fairly or very confident in their ability to help someone with a mental health problem.
• Respondents with more training (e.g., attaining MHFA trainer certification, completing multiple MHFA courses) were more likely to report being “very confident” in helping someone with a mental health problem, indicate “frequently or occasionally” engaging in helping behaviors as a result of MHFA, and endorse “agreeing or strongly agreeing” with knowing where to refer individuals for help, suggesting that the training “dose” strengthens an individual's likelihood to apply MHFA skills.
• Most survey respondents (90 percent) were correctly able to identify depression symptoms from a vignette, but performance on a general knowledge test of MHFA content was low (50 percent on average), indicating room for improvement in some areas. Consistent with this, 75 percent of respondents agreed that they could benefit from additional trainings.
• On average, a given respondent had helped approximately four individuals with a mental health problem in the past six months. Based on respondents' self-reports, altogether they helped more than 6,000 individuals who, to the best of their knowledge, ultimately sought mental health treatment.
• More than 80 percent of respondents reported using MHFA skills to support their own well-being.

City Agency

• More than half of city agency employee survey respondents indicated that they would feel comfortable discussing mental health with coworkers (65 percent) and supervisors (58 percent), which may suggest potential room for improvement with respect to workplace culture surrounding mental health. Only 11 percent would fear retaliation from their employer for seeking mental health care.
• City agency employees were significantly more likely to have recently used MHFA skills with a client than community-based trainee respondents.
• Survey respondents affiliated with education service agencies were more likely to perceive a need for more training in how to apply MHFA skills in a workplace environment than respondents from other agencies.
• Focus group participants from HRA/DSS felt that MHFA skills were critical to their work and should be part of required on-the-job training, particularly for agency staff who work directly with clients or vulnerable populations.
• To reduce burden and improve alignment with agency workflows, focus group participants from HRA/DSS felt that it would be ideal to spread out training over more than one eight-hour session; they also preferred in-person training to virtual sessions.

Community

• Most survey respondents (84 percent) reported frequently or occasionally correcting misperceptions about mental health and mental illness when they encountered them, suggesting that trainees may help to diffuse knowledge and combat stigma in their communities.
• Survey respondents identifying as Latinx/Hispanic or Black reported more frequent use of helping behaviors than their non-Hispanic White peers. They also were more likely to view MHFA training as able to address important community mental health issues and agree that they could use more MHFA training to apply skills within their community.
• In the four focus groups (three to four persons each), participants cited positive impacts of MHFA, including decreased stigma, increased mental health knowledge, and organic diffusion of skills and knowledge within their communities.
• Leaders of CBOs viewed MHFA as well-aligned with their organizational and programming priorities, and some alluded to cultural shifts within their organizations.
• Mental health stigma was identified by focus group participants as a potential barrier to community participation, though these leaders of CBOs described creative ways to sidestep this, such as building the training into other activities, offering meals during training, and having community members offer endorsements to their peers.
• Leaders from the Latinx and Chinese groups cited access to training in their primary language as a major facilitator of success for their communities. In contrast, leaders of the SGM group cited lack of cultural relevance in their MHFA training, as well as a lack of dually cultural competent trainers (LGBTQ+-informed and Spanish-speaking) as barriers to success for their community.
• When asked whether MHFA training had any potential negative impacts on their communities, community leaders did not identify any concerns.
• The length of training and the single-day format was cited by two groups (SGM and Latinx) as a possible barrier for some in their community.
• Some community organizations took steps to sustain ongoing access to MHFA training by developing their own trainings or embedded trainers within their organizations. Leaders from community-related focus groups also expressed desire to continue MHFA trainings in their communities, especially if tailored to their community (SGM leaders).

Recommendations

OCMH, DOHMH, affiliated city agencies, and NYC Opportunity should take several considerations into account as they look ahead to the future of MHFA or other mental health trainings to offer throughout New York City. Recommendations are outlined below and are accompanied with a synthesis of the survey and focus group findings that serve as the basis for these considerations.

Future Mental Health Trainings Could Be Leveraged to Address Identified Needs and to Fortify Helping Behaviors

• Mental health literacy is a potential area in need of targeting, as suggested by respondent scores on the MHFA knowledge test, which averaged 50 percent correct. Future trainings and evaluations may also benefit from efforts to refine the ways in which knowledge is measured and use of multiple measures of knowledge to clarify potential gaps and/or training targets.
• Although over three-quarters of respondents reported being very confident (30 percent) or fairly confident (47 percent) in their ability to help someone with a mental health problem, more than one in five respondents were slightly or not at all confident after participating in MHFA training. This suggests a need for additional training, and it may be beneficial for future programming to target the sources that undermine confidence in providing aid to persons in distress.
• Compared with those who had completed MHFA training more recently (within the past two years), respondents who had completed training three or more years ago were less likely to engage in certain helping behaviors (e.g., active listening, assistance with seeking professional help) but not others (e.g., providing first aid information). Moreover, participation in additional trainings was associated with more positive outcomes. Refresher trainings could be tailored to target areas that need reinforcement.
• Both survey and CBO leader focus group respondents expressed interest in future trainings that cover additional content related to behavioral health and/or that serve as boosters for MHFA training. Future implementations of MHFA or similar trainings should consider assessing trainees' specific training interests or need areas, as well as their preexisting knowledge and exposure to MHFA training or its concepts. This would promote data-driven decisions about future programming, especially among trainees who by nature of their MHFA participation are agents of diffusion with potential for wide reach within their communities.

Assess Whether MHFA Training or Similar Trainings Could Serve as a Promising Tool to Address Trainee Well-Being

• Approximately 80 percent of survey respondents reported using information from MHFA trainings to frequently or occasionally support their own well-being. Moreover, 40 percent of respondents indicated having obtained counseling as a result of MHFA training. CBO leaders and agency staff described how their own mental health needs were a motivation for taking the MHFA training and discussed how they were using information from MHFA trainings to support their well-being in daily life.
• Survey respondents presented with documented mental health needs, with more than 50 percent reporting needing help for emotional or mental health problems in the past year and 8 percent meeting criteria for serious psychological distress in the past 30 days (approximately double the rate of that found in the U.S. general population).

City Agency Employees Are More Likely to Use MHFA to Support Their Coworkers and Clients, But Workplace Mental Health Climate Can Still Be Improved

• Compared with community-based trainee respondents, city agency employee respondents were more likely to have applied MHFA to support their coworkers and clients in the past six months.
• More than 1 in 3 city agency employee respondents expressed that they would not be comfortable using mental health services through their employer or discussing mental health with coworkers or supervisors. Approximately one in ten feared retaliation or being fired for seeking mental health care.
• Agency trainees in the focus group suggested that MHFA be implemented as a mandatory training for at least some positions in their agency. The notion of mandatory training could be explored further with a range of stakeholders with distinct vantage points (e.g., city leadership, MHFA implementors, city agency leadership, city agency staff, and agency clients). The three city agency employee focus group participants felt strongly that MHFA training should be a job requirement for staff with client contact, at a minimum. Survey respondents who were required by their job to participate in MHFA training did not differ in knowledge or helping behaviors compared with their counterparts for whom MHFA training was not mandated by their employer.

MHFA Could Potentially Strengthen Social Support Networks and Community Members at Large to Serve as First-Line Supports to Individuals Experiencing Mental Health Challenges

• Respondents applied MHFA skills extensively and broadly across their social networks; 84 percent of respondents indicated using their MHFA skills to help a friend or family member, and nearly half reported applying skills with someone to whom they provide services as part of their job, a coworker, or a neighbor or acquaintance.
• Findings from the focus groups suggest that many underserved communities likely do not have sufficient resources and training to address these needs outside of clinical settings, and clinical settings have additional barriers to access (e.g., cost, cultural acceptability and accessibility, clinician workforce shortages). MHFA-trained community members may be an important first line of support, especially for peers with relatively low-level mental health needs that do not require immediate, formal clinical services.

Future Implementations of MHFA or Similar Trainings Should Consider the Impact of Community Stigma

• Approximately half of survey respondents indicated the presence of community stigma, agreeing that their community thinks less of someone with a history of mental health problems and that seeking treatment is seen as a sign of personal weakness.
• CBO leaders in the focus groups relayed that stigma was a primary motivator for facilitating MHFA training but that stigma is also a barrier to participating in MHFA training and to accessing treatment in underserved communities. Future trainings may want to explicitly consider the role of stigma as a barrier to organizational participation. For example, this might include ensuring adequate outreach and engagement with leaders in communities or neighborhoods that are known to experience higher levels of mental health-related stigma.
• Racial/ethnic minority survey respondents were more likely to report needing additional training to apply MHFA skills in their communities compared with non-Hispanic White respondents. Future trainings could assess and address the additional training needs that racial/ethnic minorities have identified as necessary to better apply MHFA within their communities.

Overall Acceptability of MHFA Training Was High, But Areas for Further Cultural Adaptation Were Identified

• Survey respondents indicated overwhelmingly favorable attitudes toward MHFA training with respect to its utility, convenience, and content (i.e., addressing issues important to the community). Furthermore, focus group participants did not identify any negative impacts of MHFA.
• Future MHFA training programs should recruit trainers who have lived experience that is shared with populations of focus. Leaders of CBOs underscored the value of offering culturally competent trainings in terms of language and lived experience. Leaders in the SGM focus group desired more culturally tailored MHFA training and culturally informed trainers.
• The use of a formal adaptation framework can provide a structured approach for adaptation that may help to preserve fidelity and effectiveness of the original intervention. The use of a formal process can provide greater scientific transparency. Formal adaptation frameworks provide a systematic, step-by-step process to identify potential adaptions to an evidence-based intervention, implement the adaptations, test the revised intervention, and implement and evaluate it on a larger scale.

Trade-Offs Between the Selection of Mental Health Programming and Desired Outcomes Should Be Weighed Carefully

• The preferred mode of delivery for MHFA training was mixed, and the city should weigh trade-offs associated with reach carefully. More data on potential trainees' preferences could be collected before future design and implementation of training initiatives, with attention to potential differences by key sociodemographic characteristics. Focus group participants had near consensus that in-person trainings were more effective and preferred, although they could identify scenarios or populations for which a virtual or hybrid training could be advantageous. At the same time, survey participants were more equivocal in terms of in-person versus online opportunities to gain additional information. One important note is that focus group participants were discussing MHFA and similar trainings specifically, while the survey assessed “additional information on mental health topics” in general. The greater acceptability of online delivery in the survey group may reflect that participants who completed the web survey might have greater comfort with and access to technology.
• Survey respondents described a range of ways they are using MHFA-associated knowledge and skills to combat stigma within their communities (e.g., 84 percent corrected misperceptions about mental health when they encountered them). CBO leaders also attested to the diffusion of MHFA knowledge and skills within their communities, which they perceived as leading to shifts in cultural norms around mental illness. Although those who had completed training three or years ago were less likely to engage in certain helping behaviors compared with those who had completed training within the past two years of the survey, the continued application of many MHFA skills among the entire sample long after the completion of training suggests that MHFA may have the potential to create longer-term sustainable approaches to altering community norms. However, more-rigorous studies are needed to establish the effectiveness of MHFA in shifting community norms around mental health. Social marketing campaigns have gained traction as a tool to counter public stigma, but reductions in stigma tend to be strongest among people who report awareness of the campaign rather than among the general population (Gaebel, Rössler, and Sartorius, 2017; Kemper and Kennedy, 2021). Nonetheless, there is some evidence that social marketing campaigns can increase perceived need for and actual mental health treatment use among individuals with psychological distress (Collins et al., 2015). When weighing the selection of mental health programming, the city should consider the strength of the evidence base for desired or prioritized outcomes (e.g., trainee knowledge, stigma, helping behaviors; city agency and community norms; perceived need and mental health service use among individuals experiencing mental health challenges).

Conclusions

Altogether, these findings suggest that MHFA may be a promising approach to building supportive social networks, organizations, and communities that are primed to recognize and come to the aid of those who need assistance with mental health challenges. The positive findings must be tempered by the fact that this evaluation is limited in its ability to causally link the self-reported outcomes to MHFA training and to establish its generalizability to the broader population of trainees. Rigorous, contemporaneous evaluation of future initiatives is critical to understanding and validating the potential effectiveness of mental health education programs like MHFA to engender impact at the individual, interpersonal, organizational, and community levels.