California's Statewide Prevention and Early Intervention (PEI) activities funded by the California Mental Health Services Authority (CalMHSA) under Proposition 63 included PEI training in mental health for staff and students in California's higher education systems. RAND evaluated a subset of these PEI trainings using an anonymous survey that asked participants to report their confidence in their ability and likelihood to refer and intervene with students with mental health issues. Participants also reported on their general satisfaction with trainings. Participants completed either a retrospective survey (reporting pre- and post-training responses in one sitting) or a pre-post survey (reporting before and after training).
Reach
RAND evaluated a portion of trainings hosted by the University of California (UC), California State University (CSU), and California Community Colleges (CCC). For UC and CSU, a portion of their standardized, manualized trainings were evaluated, including Mental Health First Aid (MHFA), Question Persuade Refer (QPR), and Applied Suicide Intervention Skills Training (ASIST). For CCC, various PEI trainings hosted by the 30 campuses with campus-based grants were evaluated. Table 1 describes the survey respondents who attended the trainings we evaluated. Across the three higher education systems, survey respondents were 74 percent female and 43 percent White, 31 percent Latino, 11 percent Asian American, 7 percent African American, and 8 percent other ethnicity.* About 63 percent were students, and others included faculty/staff as follows: 11 percent full- or part-time faculty, 8 percent administrators, 2 percent mental health or general health professionals, 12 percent other staff, and 4 percent community members (e.g., volunteers). Over 70 percent of respondents reported working with special populations, including lesbian, gay, bisexual, transgender, queer, and questioning (LGBTQ); foster care youth; and ethnic minorities.
Table 1. Sample of Evaluated Trainings
| UC | CSU | CCC | |
|---|---|---|---|
| Trainings evaluated | 99 | 436 | 93 |
| Training participants | 2,558 | 10,669 | 3,240 |
| Survey respondents [n(%)]^ | 832 (33) | 2,021 (19) | 1,208 (37) |
| Students | 398 (48) | 1,267 (63) | 728 (60) |
| Faculty/staff/admin | 434 (52) | 754 (37) | 480 (40) |
| Female | 607 (73) | 1,165 (58) | 878 (73) |
| White | 365 (45) | 872 (43) | 481 (41) |
| Latino | 164 (20) | 623 (31) | 461 (39) |
| African American | 44 (5) | 153 (8) | 70 (6) |
| Asian American | 163 (20) | 185 (9) | 73 (6) |
| Other | 84 (10) | 133 (7) | 102 (9) |
Note: Missing responses were excluded from the total count when calculating percentages.
^ Participation numbers based on estimates received from trainers, but may not reflect the true number of trainings that conducted the survey evaluation.
Training Satisfaction and Outcomes
Across systems, over 88 percent of respondents were satisified with their trainings, rating their respective training very favorably, helpful, of high quality, and important to attend. Over 81 percent of survey respondents reported that the trainings met the unique needs of the students with whom they worked. We examined whether survey respondents reported that their confidence and likelihood to refer and intervene with students changed as a result of attending the trainings. Figure 1 compares the average pre- and post-training ratings across the pre-post and retrospective surveys, with higher scores indicating greater confidence and likelihood to intervene or refer.** Respondents reported statistically significant improvements on all four outcomes after training.
Figure 1. Training Increased Reported Confidence/Likelihood of Intervening and Referring
We next examined whether respondents' ratings of training outcomes varied by race/ethnicity, role/occupation, and system (see Table 2). Latino respondents, who commonly had among the lower pre-training scores across assessed outcomes, reported significantly greater improvements than White respondents in their confidence to refer and intervene, as well as their likelihood to refer. Other ethnicities did not report significantly different responses compared with Whites. We also found that students and faculty reported significantly larger changes in their confidence and likelihood to refer, and students reported significantly larger changes in their confidence to intervene than administrators. However, health/mental health professionals, who commonly had among the highest pre-training scores, reported significantly smaller changes in their confidence to refer and intervene compared with administrators. Finally, within each system, there were statistically significant improvements from before to after the training on all four outcomes. Respondents attending CSU trainings reported larger improvements on all outcomes compared with CCC attendees, and on all outcomes except confidence to refer compared with UC attendees.
Table 2. Average Pre- and Post-Training Ratings Across Training Participants
| Confidence to Intervene | Confidence to Refer | Likelihood to Intervene | Likelihood to Refer | |||||
|---|---|---|---|---|---|---|---|---|
| Pre | Post | Pre | Post | Pre | Post | Pre | Post | |
| Race/ethnicity | ||||||||
| White | 3.6 | 4.3 | 3.7 | 4.6 | 2.9 | 3.7 | 2.8 | 3.5 |
| Latino | 3.4 | 4.3 | 3.5 | 4.5 | 2.9 | 3.7 | 2.8 | 3.6 |
| African American | 3.5 | 4.4 | 3.7 | 4.5 | 3.0 | 3.7 | 3.0 | 3.6 |
| Asian American | 3.4 | 4.2 | 3.5 | 4.4 | 2.8 | 3.6 | 2.7 | 3.4 |
| Other | 3.5 | 4.3 | 3.7 | 4.5 | 2.9 | 3.6 | 2.8 | 3.4 |
| Role/occupation | ||||||||
| Student | 3.5 | 4.3 | 3.6 | 4.5 | 2.9 | 3.6 | 2.7 | 3.5 |
| Faculty | 3.6 | 4.3 | 3.6 | 4.6 | 2.9 | 3.7 | 2.9 | 3.6 |
| Health/mental health | 4.2 | 4.7 | 4.3 | 4.8 | 3.4 | 3.9 | 3.3 | 3.7 |
| Administrator | 3.6 | 4.3 | 3.7 | 4.5 | 2.9 | 3.6 | 3.0 | 3.5 |
| System | ||||||||
| CSU | 3.5 | 4.4 | 3.6 | 4.6 | 2.9 | 3.8 | 2.8 | 3.7 |
| UC | 3.5 | 4.2 | 3.7 | 4.5 | 2.9 | 3.6 | 2.8 | 3.4 |
| CCC | 3.5 | 4.2 | 3.6 | 4.4 | 2.9 | 3.5 | 2.8 | 3.4 |
Limitations
We evaluated only a selected set of trainings in each system, and the survey responses used in the evaluation were provided by a subset of training participants and excluded data that may have been collected in other trainings. As a result, we do not know if our results are representative of all training participants, or would generalize to all PEI trainings. Response rates could have been affected by barriers to survey administration (e.g., not having time to complete the survey from home, logistical constraints such as lack of access to online survey tools). Furthermore, our outcomes are subjective (e.g., attitudes) and we do not know to what extent they would correlate with more-objective assessments of training outcomes (e.g., intervention skill). Also, while prior studies report the validity of retrospective surveys (Howard, 1980; Rockwell and Kohn, 1989; Pratt, McGuigan, and Katzev, 2000; Lam and Bengo, 2003), a “true” baseline survey was not administered prior to the training. Finally, there was no comparison group in this evaluation, which limits our understanding of the extent which our results are due specifically to the training or due to other factors.
Conclusions
Overall, these results indicate that mental health PEI trainings funded through CalMHSA reached a diverse audience, and that respondents were generally very satisfied with the trainings they attended, indicating that they were helpful and of high quality. Survey respondents attending trainings at UC, CSU, and CCC reported statistically significant improvements in their confidence to refer and intervene with students. Of note, respondents consistently reported being more confident in their ability to intervene and refer a student than they reported being likely do so, which suggests a potential need for continued efforts to increase the likelihood of individuals taking steps to intervene with and/or refer students with mental health problems. CSU respondents reported larger improvements compared with CCC and UC respondents. Further analyses are warranted to examine these system differences, including assessing whether the training content and the selection of specific trainings may have mediated the findings. The results also suggest that trainings had a larger impact on faculty and students, when compared with outcomes reported by administrators. Because faculty and students may be more likely to come into daily contact with a range of students with various strengths and needs, such a finding suggests the trainings may have a positive impact on the support of students with mental health problems. Although mental health and health professionals reported smaller improvements in their confidence to intervene and refer as a result of the trainings, their pre-training scores were higher than other occupations and likely a reflection of these individuals' general baseline level of experience, knowledge, and confidence in intervening and referring students with mental health issues. Finally, we note that respondents who self-identified as Latino reported differential gains as a result of training compared with White respondents, although Latino respondents tended to have lower pre-training scores. We do not know to what extent these differences are the result of cultural differences, variations in the characteristics of trainee participants or hosts, or other factors, but training organizers should be aware that the impact of trainings may vary across different racial/ethnic groups. In summary, our findings provide preliminary evidence that PEI trainings have been helpful in increasing participant confidence in and ability to intervene and refer, and this finding was consistent across the higher education system and across a diverse group of training participants.
References
Howard, G. S., “Response-Shift Bias a Problem in Evaluating Interventions with Pre/Post Self-Reports,” Evaluation Review, Vol. 4, No. 1, 1980, pp. 93–106.
Lam, T. C., and P. Bengo, “A Comparison of Three Retrospective Self-Reporting Methods of Measuring Change in Instructional Practice,” American Journal of Evaluation, Vol. 24, No. 1, 2003, pp. 65–80.
Pratt, C. C., W. M. McGuigan, and A. R. Katzev, “Measuring Program Outcomes: Using Retrospective Pretest Methodology,” American Journal of Evaluation, Vol. 21, No. 3, 2000, pp. 341–349.
Rockwell, S. K., and H. Kohn, “Post-Then-Pre Evaluation: Measuring Behavior Change More Accurately,” Journal of Extension, Vol. 27, No. 2, Summer 1989. As of January 5, 2015:
http://www.joe.org/joe/1989summer/a5.php
Notes
* Values may not total 100 percent because respondents could choose multiple race categories or decline to answer.
** Confidence ratings ranged from 1 (Strongly Disagree) to 5 (Strongly Agree) and likelihood ratings ranged from 1 (Not at All Likely) to 4 (Very Likely).
This research was conducted in RAND Health, a division of the RAND Corporation.
