California's Statewide Prevention and Early Intervention (PEI) activities funded by the California Mental Health Services Authority (CalMHSA) under Proposition 63 included PEI training in mental health for staff from K–12 schools in California. RAND evaluated a subset of these PEI trainings using an anonymous survey that asked participants to report, before and after the training, their confidence in their ability and likelihood to refer and intervene with students with social-emotional needs. Participants also reported on their general satisfaction with trainings. Participants completed either a retrospective survey (reporting pre- and post-training responses in one sitting) or a pre-post survey (reporting before and after training).
Reach
RAND evaluated the Training Educators Through Recognition and Identification Strategies (TETRIS) training sponsored by the California Department of Education (CDE), delivered by the Placer County Office of Education, and hosted by county offices of education (COE), and also evaluated select PEI trainings sponsored by the California County Superintendents Educational Services Association (CCSESA). The CDE-sponsored TETRIS training lasts about eight hours and focuses on increasing knowledge of student risk and protective factors, school and community resources, intervention strategies, and ways to promote learning environments that foster mental health and wellness. The CCSESA trainings varied in duration and included topics such as suicide prevention, mental health promotion, bullying prevention, and other trainings focusing on promoting protective behaviors and decreasing problem behaviors; some of which also included trainings with pre-K/preschool teachers and staff. RAND evaluated all eight TETRIS trainings conducted in 2013–2014 (n = 402 participants; mean = 50 participants per training; 84-percent survey response rate), and 834 CCSESA trainings (n = 26,343 participants; mean = 31 participants per training; 74-percent survey response rate). Table 1 describes the survey respondents who attended the trainings we evaluated.
Table 1. Sample of Evaluated Trainings
| CDE | CCSESA | |
|---|---|---|
| Trainings evaluated | 8 | 834 |
| Training participants | 402 | 26,343 |
| Survey respondents [n(%)] | 337 (84) | 19,588 (74) |
| Female | 264 (80) | 12,741 (76) |
| White | 143 (45) | 8,282 (45) |
| Latino | 101 (32) | 7,285 (40) |
| African American | 26 (8) | 894 (5) |
| Asian American | 25 (8) | 987 (5) |
| Other | 22 (7) | 836 (5) |
NOTE: Missing responses were excluded from the total count when calculating percentages.
Across the CDE- and CCSESA-sponsored trainings, survey respondents were 76 percent female; 45 percent White, 40 percent Latino, 5 percent Asian American, 5 percent African American, and 5 percent other ethnicity.* About 52 percent were full- or part-time teachers, 17 percent mental health or general health professionals, 15 percent administrators, 7 percent other staff, and 8 percent community members (e.g., volunteers). Over 71 percent of respondents reported working with special populations, including lesbian, gay, bisexual, transgender, queer, and questioning (LGBTQ); foster care youth; and ethnic minorities.
Training Satisfaction and Outcomes
Over 84 percent of respondents were satified with the trainings, rating their respective trainings very favorably, helpful, and important to attend. Almost 85 percent of survey respondents reported that the trainings met the unique needs of the students with whom they worked.
We examined whether survey respondents reported that their confidence and likelihood to refer and intervene with students with social-emotional needs changed as a result of attending the trainings. Figure 1 compares the average pre- and post-training ratings across the pre-post and retrospective surveys, with higher scores indicating greater confidence and likelihood.** Respondents generally reported higher confidence to refer ratings compared with likelihood to intervene ratings, and statistically significant improvements on all four outcomes after training.
Figure 1. Training Increased Reported Confidence/Likelihood of Intervening and Referring
We next examined whether respondents' outcome ratings varied by role/occupation, race/ethnicity, and sponsoring system across all trainings (i.e., CCSESA or CDE; see Table 2). Full- and part-time teachers and other staff (e.g., secretaries, after-school program staff, program specialists), who commonly had lower pre-training scores, reported significantly larger changes in their confidence and likelihood to refer and intervene when compared with administrators. In contrast, health and mental health professionals, who commonly had higher pre-training scores, reported significantly smaller changes in their confidence and likelihood to refer and intervene compared with administrators. Trainees who self-identified as Asian American, African American, Latino, and other all reported significantly larger improvements on all four outcomes (confidence and likelihood to intervene and refer) when compared with White respondents. Respondents from CDE and CCSESA trainings reported similar improvements in their confidence to intervene and likelihood to refer, with participants from CCSESA reporting significantly larger improvements in their confidence to refer, and CDE training participants reporting significantly greater improvements in their likelihood to intervene with students with emotional or behavioral problems.
Table 2. Average Pre- and Post-Training Ratings Across Training Participants
| Confidence to Intervene | Confidence to Refer | Likelihood to Intervene | Likelihood to Refer | |||||
|---|---|---|---|---|---|---|---|---|
| Pre | Post | Pre | Post | Pre | Post | Pre | Post | |
| Role/occupation | ||||||||
| Teachers | 3.6 | 4.1 | 3.5 | 4.2 | 2.9 | 3.4 | 3.1 | 3.5 |
| Health/Mental Health | 4.1 | 4.5 | 4.0 | 4.5 | 3.5 | 3.8 | 3.5 | 3.8 |
| Other staff | 3.6 | 4.2 | 3.6 | 4.3 | 2.9 | 3.5 | 3.1 | 3.6 |
| Administrator | 3.9 | 4.4 | 3.8 | 4.4 | 3.2 | 3.7 | 3.4 | 3.7 |
| Ethnicity | ||||||||
| White | 3.7 | 4.2 | 3.6 | 4.3 | 3.1 | 3.5 | 3.2 | 3.6 |
| Latino | 3.7 | 4.3 | 3.6 | 4.3 | 3.1 | 3.6 | 3.2 | 3.7 |
| African American | 3.7 | 4.3 | 3.6 | 4.4 | 3.1 | 3.6 | 3.2 | 3.7 |
| Asian American | 3.6 | 4.2 | 3.6 | 4.3 | 3.1 | 3.6 | 3.2 | 3.6 |
| Other | 3.7 | 4.2 | 3.6 | 4.3 | 3.1 | 3.5 | 3.2 | 3.6 |
| System | ||||||||
| CCSESA | 3.7 | 4.2 | 3.6 | 4.3 | 3.1 | 3.5 | 3.2 | 3.6 |
| CDE | 4.0 | 4.5 | 3.9 | 4.5 | 3.2 | 3.7 | 3.3 | 3.7 |
Limitations
We evaluated only a selected set of trainings, and the survey responses used in the evaluation were provided by a subset of training participants. As a result, we do not know if our results are representative of all training participants, or would generalize to all PEI trainings. Furthermore, our outcomes are subjective (e.g., attitudes) and we do not know to what extent they would correlate with more-objective assessments of training outcomes (e.g., intervention skill). Also, while prior studies report the validity of retrospective surveys (Howard, 1980; Rockwell and Kohn, 1989; Pratt, McGuigan, and Katzev, 2000; Lam and Bengo, 2003), a “true” baseline survey was not administered prior to the training. Finally, there was no comparison group in this evaluation, which limits our understanding of the extent to which our results are due specifically to the training or due to other factors.
Conclusions
Overall, these results indicate that mental health PEI trainings funded through CalMHSA reached a diverse audience, and that respondents were generally very satisfied with the trainings they attended, indicating that they were helpful and served respondents' student population. Survey respondents reported statistically significant improvements in their confidence to refer and intervene with K–12 students who appeared to be emotionally distressed. Of note, respondents consistently reported being more confident in their ability to intervene and refer a student than they reported being likely do so, which suggests a potential need for continued efforts to increase the likelihood of individuals taking steps to intervene with and/or refer students with social-emotional needs. These results also suggest that trainings had a larger impact on teachers and other staff members, when compared with outcomes reported by administrators. Because teachers may be more likely to come into daily contact with a range of students with various strengths and needs, such a finding suggests the trainings may have a positive impact on the support of students with social-emotional needs. Although mental health and health professionals reported smaller improvements in their confidence to intervene and refer as a result of the trainings, their pre-training scores were higher than other occupations and likely a reflection of these individuals' general baseline level of experience, knowledge, and confidence in intervening and referring students with mental health issues. Finally, we note that several racial/ethnic subgroups (Asian American, African American, Latino, and other) reported greater improvements as a result of trainings compared with White respondents. While these differences were statistically significant, the magnitude of change was similar across race/ethnicity, and further evaluation would be needed to determine if the differences were associated with how individuals in various groups behaved. We do not know to what extent these differences are the result of cultural difference, variations in the characteristics of trainee participants or hosts, or other factors, but training organizers should be aware that the impact of trainings may vary across different racial/ethnic groups. In summary, our findings provide preliminary evidence that PEI K–12 trainings have been helpful in increasing participant confidence in and ability to intervene and refer, and this finding was consistent across the K–12 grades and across a diverse group of training participants.
References
Howard, G. S., “Response-Shift Bias a Problem in Evaluating Interventions with Pre/Post Self-Reports,” Evaluation Review, Vol. 4, No. 1, 1980, pp. 93–106.
Lam, T. C., and P. Bengo, “A Comparison of Three Retrospective Self-Reporting Methods of Measuring Change in Instructional Practice,” American Journal of Evaluation, Vol. 24, No. 1, 2003, pp. 65–80.
Pratt, C. C., W. M. McGuigan, and A. R. Katzev, “Measuring Program Outcomes: Using Retrospective Pretest Methodology,” American Journal of Evaluation, Vol. 21, No. 3, 2000, pp. 341–349.
Rockwell, S. K., and H. Kohn, “Post-Then-Pre Evaluation: Measuring Behavior Change More Accurately,” Journal of Extension, Vol. 27, No. 2, Summer 1989. As of January 5, 2015:
http://www.joe.org/joe/1989summer/a5.php
Notes
* Values may not total 100 percent because respondents could choose multiple race categories or decline to answer.
** Confidence ratings ranged from 1 (Strongly Disagree) to 5 (Strongly Agree) and likelihood ratings ranged from 1 (Not at All Likely) to 4 (Very Likely).
This research was conducted in RAND Health, a division of the RAND Corporation.
