A Case Study Evaluating the Fidelity of Suicide Prevention Workshops in California

by Karen Chan Osilla, Dionne Barnes-Proby, Marylou Gilbert, Rajeev Ramchand

This Article

RAND Health Quarterly, 2014; 4(3):11

Abstract

Reports on an evaluation of California suicide intervention training workshops to provide evidence that training is delivered with fidelity and adherence to the workshop’s design, as well as with high quality.

For more information, see RAND RR-755-CMHSA at https://www.rand.org/pubs/research_reports/RR755.html

Full Text

A critical goal of the California Mental Health Services Authority's (CalMHSA's) Statewide Prevention and Early Intervention (PEI) activities funded under Proposition 63 is dissemination of suicide intervention training to individuals in the community. In one training component, the organization LivingWorks was contracted to provide Training-for-Trainers (T4T) for Applied Suicide Intervention Skills Training (ASIST). Ten T4T workshops were conducted to teach individuals to be ASIST trainers so they could disseminate ASIST workshops to colleagues and other individuals in the community. The ASIST workshops spanned two days and taught participants “suicide first aid” so they can recognize those at risk of suicide and intervene.

To understand how policies improve suicide prevention efforts, it is important to evaluate the fidelity and adherence of workshops because prevention funding is scarce and needs to be dispersed wisely and monitored closely. In addition, providing feedback to newly trained trainers may help ensure that workshops are delivered with high quality. As part of RAND's evaluation of CalMHSA's Statewide PEI activities, the evaluation team observed five ASIST workshops delivered by trainers who had recently attended one of the T4Ts. We used a convenience sample of individuals who received Proposition 63–funded ASIST T4T but who were not yet experienced trainers. We asked newly trained trainers in different regions of California who served diverse populations whether we could observe their workshops. Of the trainers asked, all agreed to be observed. The goal of the evaluation team's attendance at the workshops was to provide a preliminary assessment of whether trainers were delivering their ASIST workshops with high fidelity (i.e., covering all aspects of the manual-based workshop) and adherence (i.e., whether the workshop followed the recommended style of presenting) to provide evidence that workshops were implemented as intended and with high quality. The five workshops observed were held throughout the state, were targeted to different communities (i.e., law enforcement, military, mental health and crisis workers), and were conducted in both English and Spanish. The number of participants per workshop ranged from seven to 35 people. In total, across the five workshops, 115 participants received the workshop (39 men and 76 women).

Together with LivingWorks, RAND developed a fidelity and adherence tool that could reliably be used when two members of the evaluation team attended workshops. There were 59 fidelity items that captured whether trainers covered specific sections of the ASIST protocol and an additional 16 items that assessed whether trainers adhered to the recommended ASIST presentation style (see Table 1).

Table 1. ASIST Presentation Style Adherence

Adherence Item Trainers (%)*
ASIST trainer competencies
Suicide mentioned specifically 100
Positive feedback to participants 10
No negative feedback to participants 80
Worked within Suicide Intervention Model framework 60
General facilitator proficiencies
Collaborative with participants 50
Open-ended questions 40
Well-organized simulations 30
Conveyed empathy 60
Group management skills 80
Overall group participation 80
Role play and discussion participation 80
Time management skills 40
Tailored concepts to target population 20
Accepting of diverse cultural differences 60
Knowledgeable about cultural beliefs 20
Acknowledged participants' experiences 40

* Trainers who scored as adherent “most of the time” or “all of the time.”

Fidelity. In three of the five workshops, trainers covered 75 percent or more of the fidelity items, which demonstrates thorough review of the workshop content (Cross et al., 2014). In the other two workshops, trainers covered roughly two-thirds of workshop content.

Adherence. Trainers generally adhered to one of the four ASIST competencies and five of the 11 general competencies relating to group management. However, the trainers may need to improve their efforts to tailor content to specific audiences, promote cultural competence, and manage time.

The results from these five workshops should not be generalized to the delivery of all ASIST workshops. This component of our evaluation provided a tool that could be used for continued quality monitoring and improvement of ASIST workshops throughout the state. Overall, data from select workshops show that trainers in this evaluation tended to score higher in fidelity than adherence. These results provide useful insights into areas in which these trainers could improve. Anecdotally, we know from our previous research that it takes time for trainers to learn a protocol. Initially, trainers tend to read the protocol from their manuals and are not as engaging and collaborative with workshop participants because they are focused on covering the material. However, time to practice delivering more workshops may not be sufficient to increase fidelity and adherence (Cross et al., 2014). Trainers may also need standardized and concrete feedback to hone specific skills. The tool developed for this study can facilitate this. We suggest that experienced ASIST trainers use the fidelity and adherence protocol to guide post-workshop discussions with the newer trainers with whom they are co-leading ASIST workshops. In addition, participants of workshops themselves can be given the fidelity and adherence tool to evaluate whether all prescribed workshop components were covered and the style in which the information was conveyed. It is essential to provide feedback to new trainers so they can improve in concrete ways and, ultimately, deliver ASIST as intended by the developers and with high quality.

References

Cross, W. F., A. R. Pisani, K. Schmeelk-Cone, Y. Xia, X. Tu, J. L. Munfakh, and M. S. Gould, “Measuring Trainer Fidelity in the Transfer of Suicide Prevention Training,” Crisis, Vol. 35, No. 3, 2014 (in press), pp. 202–212.

This research was conducted in RAND Health, a division of the RAND Corporation.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.