A Preliminary Evaluation of Interactive Video Simulation Training for Campus Law Enforcement in California

by Courtney Ann Kase, Karen Chan Osilla, Rachana Seelam, Michelle W. Woodbridge, Bradley D. Stein

This Article

RAND Health Quarterly, 2016; 6(2):4

Abstract

This article evaluates the Interactive Video Simulation Training to help campus law enforcement professionals refer and intervene with college students experiencing psychological distress.

For more information, see RAND RR-1389-CMHSA at https://www.rand.org/pubs/research_reports/RR1389.html

Full Text

Campus law enforcement professionals are tasked with promoting and maintaining a safe campus environment. They are first responders and are often the first professionals in a position to help support and assist a student experiencing psychological distress, or a student experiencing an event or situation that puts him or her at risk for psychological distress. Mental health problems are common among students. In one survey, nineteen percent of students in California's higher education system reported experiencing serious psychological distress in the past 30 days, and 11 percent reported significant mental health–related academic impairment in the past year. Additionally, 11 percent of students reported feeling hopeless most or all of the time (Sontag-Padilla et al., 2014). Campus law enforcement professionals with the necessary knowledge and skills can play a particularly important role in identifying students experiencing psychological distress, consulting with mental health personnel, and referring students to mental health supports and services when responding to events involving members of a campus community (Margolis and Shtull, 2012; Reuland and Margolis, 2003; Martinez, 2012; Hails and Borum, 2003).

To enhance the skills of campus law enforcement professionals, statewide Prevention and Early Intervention (PEI) activities funded by California counties through the California Mental Health Services Authority (CalMHSA) under Proposition 63 included PEI training in mental health for law enforcement professionals in California's higher education systems. In June and July 2014, the California State University (CSU) system hosted 14 four-hour Interactive Video Simulation Training (IVST) sessions at five locations for CSU law enforcement professionals, law enforcement professionals from University of California and California Community College campuses, and local police agencies. Attendees represented more than 28 different law enforcement agencies from higher education campuses and communities surrounding such campuses. The IVST sessions use an innovative portable simulator, which enables law enforcement professionals to interact in real time with life-size, video-projected subjects who role-play individuals experiencing psychological distress. Participants make step-by-step decisions on how best to respond to the presented situation. In addition to the IVST sessions, participants are taught how to recognize the signs of mental health problems, deescalate conflict, and provide on-campus and off-campus referrals for mental health services (Office of the Chancellor, California State University, 2014; Santa Clara County Mental Health Department, 2014). The training aims to provide campus law enforcement professionals with the knowledge and skills to help in the effective identification of, assessment of, and intervention with students experiencing psychological distress (Council of State Governments, 2002; Hails and Borum, 2003; Margolis and Shtull, 2012; Reuland and Schwarzfield, 2008; Schwarzfield, Reuland and Plotkin, 2008; Office of the Chancellor, California State University, 2014). This report describes a preliminary evaluation of the 14 IVST sessions to measure participants' perceptions of the training's usefulness and quality, as well as changes in participants' self-reported (1) confidence to refer, (2) confidence to intervene, (3) likelihood to refer, and (4) likelihood to intervene if encountering a student experiencing psychological distress.

Methods

Eighty-eight percent (n = 192) of participants in the IVST trainings completed a retrospective paper or online survey immediately after the training. The survey assessed participants' pre- and post-training self-reported confidence and likelihood to refer and intervene with students experiencing psychological distress.

Measures

To measure the impact of training on participants' self-reported confidence and likelihood to refer, as well as their confidence and likelihood to intervene with students, we adapted items from the Suicide Prevention Survey (Wyman et al., 2008). This measure was designed for evaluations of the Question, Persuade, and Refer (QPR) gatekeeper-training program in high schools and colleges (Tompkins and Witt, 2009; Wyman et al., 2008).

We used four items to assess training usefulness and quality, each along a five-point Likert scale, with lower scores indicating lower perceived importance or dissatisfaction with training. Items included, “Please rate the quality of training that you received,” and “It is important for law enforcement and related staff to attend trainings like this one to support students/youth with mental health problems.”

We measured individuals' self-reported confidence to refer and confidence to intervene using five items, each along a five-point Likert scale (1 = strongly disagree and 5 = strongly agree) (measures adapted from Tomkins and Witt, 2009; Wyman et al., 2008). For example, items to measure self-reported confidence in ability to refer included, “I can identify the places or people where I should refer students/youth with mental health needs/distress.”

We measured participants' self-reported likelihood to refer and likelihood to intervene using nine items along a four-point Likert scale (1 = not at all likely and 4 = very likely) (measures adapted from Tompkins and Witt, 2009; Shaffer et al., 1991). Each item began with, “If a student showed signs that s/he might be experiencing psychological distress, I would . . .” and was followed by such prompts as, “Encourage him/her to get professional help (e.g., hospital, mental health center, counselor, etc.).”

Analyses

Paired t-tests were used to assess changes between pre- and post-training scores and to determine whether the pre- and post-training mean outcomes varied significantly (p < 0.05).

Results

Participating training respondents were 84 percent male (n = 162). They identified as 31 percent Latino (n = 58); 44 percent white, not Latino (n = 83); 10 percent African-American (n = 18); 8 percent Asian (n = 16); and 7 percent other (n = 13). Respondents' roles were identified as 81 percent police officer (n = 155), 12 percent other security (n = 22), < 1 percent counselor (n = 1), and 7 percent other staff (n = 13). About 85 percent of respondents reported currently working with undergraduate students, and about 58 percent reported currently working with graduate students.

Perceptions of the Training

Ninety-six percent of the participants reported that attending the training was somewhat or very helpful and agreed that it was important for law enforcement professionals to attend trainings to support students with mental health problems. Nearly 90 percent of participants reported that the training met the needs of the students or youth they interact with on campus (see Table 1).

Table 1. Favorable Perceptions of the Training

Survey Item Percentage Who Agreed or Strongly Agreed
The quality of training received was good or excellent. 96
The training received was helpful. 96
“The training meets the unique needs of the students/youth I come in contact with (e.g., diverse ethnic/language groups, Lesbian, Gay, Bisexual, Transgender, Queer or Questioning (LGBTQ), low income).” 89
“It is important for law enforcement and related staff to attend trainings like this one to support students/youth with mental health problems.” 96

Changes in Confidence and Likelihood to Refer and Intervene as Result of Training

Table 2 shows the average pre- and post-training scores on participants' confidence and likelihood to refer and intervene and the results of the paired t-test to determine whether beliefs differed significantly from pre- to post-training. Participants reported significant improvements on all four outcomes. After attending the training, participants expressed greater confidence and likelihood to refer students with mental health problems to resources, as well as greater confidence and likelihood to intervene if a student showed signs of psychological distress.

Table 2. Significant Changes on All Measures from Pre- to Post-Training

Measure Subscales Pre-Training Mean (SD) Post-Training Mean (SD) Paired T-Test P-Value
Confidence to refera 3.84 (0.72) 4.43 (0.57) <0.0001
Confidence to intervenea 4.00 (0.76) 4.51 (0.55) <0.0001
Likelihood to referb 3.18 (0.64) 3.61 (0.54) <0.0001
Likelihood to interveneb 3.34 (0.65) 3.80 (0.38) <0.0001

NOTE: SD = standard deviation.

a. Five-point scale: 1 = strongly disagree, 2 = disagree, 3 = neither agree or disagree, 4 = agree, 5 = strongly agree.

b. Four-point scale: 1 = not at all likely, 2 = somewhat likely, 3 = likely, 4 = very likely.

Discussion

We evaluated law enforcement professionals' ratings of a sample of IVST training sessions for referring and intervening with students experiencing psychological distress and acting as gatekeepers to the metal health system. We assessed how participants' self-reported confidence and likelihood of identifying, assessing, and intervening to support students experiencing psychological distress changed post-training. Participants reported finding the sessions helpful and agreed that the training was important for them and their peers to attend. Participants indicated increased confidence and likelihood to assist a student or youth experiencing psychological distress. These findings are similar to recently reported outcomes of PEI trainings among staff and students in the state's higher education system (Osilla et al., 2015).

These findings should be considered in light of study limitations. Our outcomes are subjective, because we asked about participants' self-reported confidence and likelihood to refer and intervene, and we do not know to what extent the results would correlate with more-objective assessments of training outcomes, such as skill or actual behavior post-training. Also, although prior studies report the validity of retrospective surveys (Rockwell and Kohn, 1989; Rohs, 1999; Pratt, McGuigan, and Katzev, 2000; Lam and Bengo, 2003), a “true” baseline survey was not administered prior to the training. Finally, we examined outcomes only immediately after the training; whether the positive shifts in knowledge and attitudes are maintained over the longer term is unknown, and further investigation is warranted. Nonetheless, our findings indicate that IVST, as presented in the settings included within this evaluation, can result in immediate improvements in law enforcement professionals' confidence and likelihood to refer students to mental health resources and services, as well as confidence and likelihood to intervene if a student shows signs of experiencing psychological distress. If campus law enforcement professionals are first responders to students experiencing psychological distress, and if higher education institutions want them to respond appropriately and effectively, then they need the knowledge and skills to carry out these tasks. More research is needed to better understand how to equip law enforcement to identify students in psychological stress, how IVST compares with other trainings, and whether skills learned during the training correlate with actual changes in behavior.

References

Council of State Governments, Criminal Justice/Mental Health Consensus Project, Lexington, Ky., June 2002.

Hails, J., and R. Borum, “Police Training and Specialized Approaches for Responding to People with Mental Illness,” Crime and Delinquency, Vol. 49, No. 1, January 2003, pp. 52–61.

Lam, T. C., and P. Bengo, “A Comparison of Three Retrospective Self-Reporting Methods of Measuring Change in Instructional Practice,” American Journal of Evaluation, Vol. 24, No. 1, 2003, pp. 65–80.

Margolis, G. J., and P. R. Shtull, “The Police Response to Mental Illness on Campus,” Journal of College Student Psychotherapy, Vol. 26, No. 4, 2012, pp. 307–321.

Martinez, L. E., “The Police Response to Critical Incidents in Academic Institutions,” Journal of Police Crisis Negotiations, Vol. 12, No. 1, 2012, pp. 69–77.

Office of the Chancellor, California State University, “CSU to Lead Mental Health Training for Law Enforcement Officers,” Humboldt State University, June 6, 2014. As of July 13, 2015:
http://now.humboldt.edu/news/csu-to-lead-mental-health-training-for-law-enforcement-officers/

Osilla, K. C, M. W. Woodbridge, R. Seelam, C. A. Kase, E. Roth, and B. D. Stein, Mental Health Trainings in California's Higher Education System Are Associated with Increased Confidence and Likelihood to Intervene with and Refer Students, Santa Monica, Calif.: RAND Corporation, RR-954-CMHSA, 2015. As of May 5, 2016:
http://www.rand.org/pubs/research_reports/RR954

Pratt, C. C., W. M. McGuigan, and A. R. Katzev, “Measuring Program Outcomes: Using Retrospective Pretest Methodology,” American Journal of Evaluation, Vol. 21, 2000, pp. 341–349.

Reuland, M., and G. J. Margolis, “Police Approaches That Improve the Response to People with Mental Illnesses: A Focus on Victims,” The Police Chief, Vol. 70, No. 11, November 2003. As of May 5, 2016:
http://www.policechiefmagazine.com/magazine/index.cfm?fuseaction=display&article_id=145&issue_id=112003

Reuland, M., and M. Schwarzfield, Improving Responses to People with Mental Illness: Strategies for Effective Law Enforcement Training, New York: Council of State Governments, Justice Center, 2008.

Rockwell, S. K., and H. Kohn, “Post-Then-Pre Evaluation,” Journal of Extension, Vol. 27, No. 2, 1989, pp. 19–21.

Rohs, F. R., “Response Shift Bias: A Problem in Evaluating Leadership Development with Self-Report Pretest-Posttest Measures,” Journal of Agricultural Education, Vol. 40, No. 4, 1999, pp. 28–37.

Santa Clara County Mental Health Department, Final Evaluation of Innovation #8: Interactive Video Simulation Training, Oakland, Calif.: Resource Development Associates, 2014. As of May 5, 2016:
https://www.sccgov.org/sites/mhd/MHSA/Documents/SLC%20NEW/July%2029,%202014%20Meeting%20Materials/INN-08%20Interactive%20Video%20Simulation%20Training%20Project%20Evaluation%20Report.pdf

Schwarzfield, M., M. Reuland, and M. Plotkin, Improving Responses to People with Mental Illness: The Essential Elements of a Specialized Law Enforcement-Based Program, New York: Council of State Governments, Justice Center, 2008.

Shaffer, D., A. Garland, V. Vieland, M. Underwood, and C. Busner, “The Impact of Curriculum-Based Suicide Prevention Programs for Teenagers,” Journal of American Academy of Child and Adolescent Psychiatry, Vol. 30, No. 4, 1991, pp. 588–596.

Sontag-Padilla, L., E. Roth, M. W. Woodbridge, C. A. Kase, K. C. Osilla, E. J. D'Amico, L. H. Jaycox, and B. D. Stein, CalMHSA Student Mental Health Campus-Wide Survey: 2013 Summary Report, Santa Monica, Calif.: RAND Corporation, RR-685-CMHSA, 2014. As of May 5, 2016:
http://www.rand.org/pubs/research_reports/RR685

Tompkins, T. L., and J. Witt, “The Short-Term Effectiveness of a Suicide Prevention Gatekeeper Training Program in a College Setting with Residence Life Advisers,” Journal of Primary Prevention, Vol. 30, No. 2, 2009, pp. 131–149.

Wyman, P. A., C. Hendricks Brown, J. Inman, W. Cross, K. Schmeelk-Cone, J. Guo, and J. B. Pena, “Randomized Trial of a Gatekeeper Program for Suicide Prevention: 1-Year Impact on Secondary School Staff,” Journal of Consulting and Clinical Psychology, Vol. 76, No. 1, 2008, pp. 104–115.

This research was conducted in RAND Health, a division of the RAND Corporation.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.