The California Mental Health Services Authority's (CalMHSA's) statewide prevention and early intervention initiative funded 12 crisis call centers to create, expand, or enhance their services related to suicide prevention. RAND conducted live monitoring of calls made to ten of 12 call centers as part of an evaluation of the suicide prevention funding. We designed this protocol for silent monitoring of incoming crisis calls, building on earlier work by other researchers. The protocol was designed with the following features:
- Assesses domains relevant to call content and quality
- Builds on prior work for potential comparability to national studies
- Allows for rapid coding and reliability across raters
- Might be useful for ongoing quality assurance monitoring beyond this study.
Designed for a diverse set of crisis call centers in California to assess aspects of best practice as well as to describe calls, the protocol may be useful for the evaluation of other call centers or for continuous quality improvement within call centers. Specifically, call centers that already use silent monitoring or record calls by supervisors or trainers may find the protocol useful for documenting strengths and weaknesses in their staff members or in the call center as a whole, helping to identify areas for further training. In addition, documentation of the types of calls received may help in planning future trainings, funding opportunities, and development of referrals for the most common needs among callers. Call centers that do not use silent monitoring or record calls may find the protocol useful for telephone counselor self-assessment.
In the remainder of this document, we describe our methods for developing and testing the protocol. Then, we describe our rationale for each of the items included in the protocol and present two tables that briefly summarize past evaluations of crisis lines. The protocol itself can be found via PDF link at the end of this document.
Methods of Protocol Development
The design of this silent monitoring protocol began with a review of the literature of other studies that involved ratings of crisis line calls, and a review of protocols developed in those studies or currently in use by the researchers. A summary of this research is available in Table 1.
In addition, we consulted extensively with Dr. Madelyn Gould, who has used silent monitoring for research extensively in the past (Gould et al., 2013; Gould et al., 2007; Gould et al., 2012; Kalafat et al., 2007). She shared two of her past protocols with us (Gould et al., 2013; Gould et al., 2012) and discussed implementation issues as well as her team's findings to date. We used portions of her protocols but streamlined and simplified many parts to fit feasibility constraints in the CalMHSA evaluation as well as potential future use by crisis call centers for continuous quality improvement.
Next, we visited a National Suicide Prevention Lifeline (NSPL) call center and listened to taped crisis calls to pilot test our protocol. This resulted in further streamlining and refinement. We visited a second center for training and refinement purposes. At this visit, the two individuals tasked with conducting all silent monitoring for this study double-coded taped crisis calls. All authors then discussed areas of confusion and made adjustments to the protocol. For instance, there were many more third-party and non-crisis informational calls than expected, so the protocol was adjusted to describe those calls more fully. Finally, two raters visited one of the CalMHSA crisis call centers to double-code 18 calls to examine reliability of coding (results presented in Table 1 and Table 2). We used the results to further refine the protocol and its instructions to improve reliability.
Section A: Call Characteristics
This section was designed to describe basic aspects of the call itself, including its eligibility for silent monitoring. The section was modified from both Gould protocols (Gould et al., 2013; Gould et al., 2012). Our modifications included dropping some questions that did not apply to our study, tailoring the eligibility criteria, and adding a question that allowed us to document that the silent monitor had difficulty hearing the call. The final protocol included one demographic question (gender), call characteristics (start and end time, technical issues that resulted in abrupt termination of the call, difficulty hearing the call, and whether the caller was put on hold), type of call (continuation of previous call with same counselor, repeat caller), and reasons for ineligibility for monitoring (caller under 18 years old, counselor not participating in evaluation, non–English-speaking caller, obscene/prank caller). A call selection status item was created to summarize these ineligibility criteria and indicates whether the call was included in the evaluation.
Section B: Call Content
This section focused on the types of issues and problems raised during the call. In the case of calls made by individuals concerned about somebody else (i.e., third-party callers), this could include both problems for the caller and for the person they were calling about. We chose the topic areas by drawing from the list of presenting problems and stressful experiences on both Gould protocols (Gould et al., 2013; Gould et al., 2012), but added the assessment of problems “for someone else” to capture third-party call content. Problems listed included mental health problems, substance abuse, homelessness, relationship problems, exposure to violence or abuse, and suicidal ideation or intent. We included an “other” category to capture other important content not listed. Monitors were trained to check items regardless of whether the caller mentioned them spontaneously or call responders asked questions to bring them to light, and that these items did not have to be the reason for the call in order to be checked.
Section C: Suicide Risk Assessment
This section was designed to assess adherence to NSPL guidelines that call for suicide assessment for every crisis center call, as well as the types of issues explored for those callers who expressed suicidal ideation or intent for themselves or a third party. These guidelines suggest that telephone responders should ask a minimum of three questions to determine suicide risk, unless the caller offers the information spontaneously: (1) Are you currently thinking of suicide? (current ideation); (2) Have you thought about suicide in the past two months? (recent ideation); and (3) Have you ever attempted to kill yourself? (attempt). It is recommended that all three questions be asked of all callers, regardless of how the caller responds to the other suicide risk questions, to provide a sense of callers' emotional instability as well as to build rapport (Joiner et al., 2007).
Items in this section were modified from the Gould Applied Suicide Intervention Skills Training (ASIST) protocol (Gould et al., 2013) to match the wording from the NSPL for the questions to ask each caller. The questions were expanded to include exploration of suicidal ideation or intent. For each question, the monitors also indicate whether the information was offered spontaneously by the caller or asked by the call responder. If offered spontaneously, the monitor indicates whether the call responder asked follow-up questions to explore the disclosure.
We also added two items pertinent to suicide assessment regarding individuals who had already taken steps to harm themselves (or that a third party took to harm him or herself) prior to the call: preparatory behaviors or an attempt in progress. This information appeared to be missing, and we deemed it necessary to gauge imminent risk in the next section.
In addition, this section contains information about the degree to which four risk factors related to suicide are explored on the call. These include suicide desires, suicide capability, suicide intent, and lack of buffers/connectedness. These items were based on NSPL guidelines for suicide assessment (Joiner et al., 2007) and draw from the Lifeline Quality Improvement Monitoring Tool (Lifeline QI Monitoring Tool). Monitors were trained to indicate that a risk factor was assessed if it was mentioned on the call, regardless of which party introduced it.
Section D: Imminent Risk
This section begins with a global assessment of whether imminent risk of harm to the caller or third party was present at any point during the call, similar to an item in the Gould ASIST protocol (Gould et al., 2013) and the Lifeline QI Monitoring Tool but expanded to include third parties. If imminent risk was deemed to be present, another item assessed whether a rescue was initiated or whether risk was reduced enough during the call such that a rescue was not necessary. We also added items to indicate other types of imminent risk, including imminent threat of violence or impairment due to substance use, to capture the types of calls we sometimes heard when developing the protocol. Following the Gould ASIST protocol (Gould et al., 2013), the silent monitors had the ability to talk to a supervisor if they noted imminent risk that was not resolved by the end of the call. These circumstances were also recorded in this section, along with any reasons for not initiating a rescue.
Section E: Action Plan for Imminent Risk
This section was included only for those calls that were deemed to contain some element of imminent risk, whether for suicide or something else. In this section, an action plan is defined as a suggested course of action to alleviate distress or increase safety. The types of questions were drawn from multiple sources (Gould et al., 2013; Gould et al., 2012), including the Lifeline QI Monitoring Tool, but simplified considerably for this protocol. They include specific plans for self-help, increasing safety, increasing social interactions, and seeking professional help, as well as discussion about logistics and potential barriers to the plan, to increase the probability that it will be followed. A final question asks about the extent to which the caller agreed to the plan.
Section F: Telephone Counselor Response
This section addressed the types of actions and behaviors that call responders engaged in. Items were adapted from the Gould ASIST protocol (Gould et al., 2013), but the response options were modified to improve reliability across raters. Behaviors assessed include positive responses, such as reflective listening and sensitivity, as well as negative responses, such as criticism or judging.
Section G: Changes During Call
Two items assessed the level of caller distress at the beginning of the call and at the end of the call. This is similar to Mishara's study of call outcomes (Mishara, Chagnon, and Daigle, 2007b) but simplified into a single global distress item rather than items on several emotional distress dimensions (e.g., sad/happy, agitated, desperate). We added a rating for third-party calls as well.
Section H: Overall Ratings
This section required subjective ratings on the overall content of the call and how the call ended. We based these items on several sources (Gould et al., 2013; Gould et al., 2012) and the Lifeline QI Monitoring Tool but simplified them into seven main areas: good contact/rapport, problem-solving, referrals, an overall effectiveness rating for each of three types of calls (urgent, distressed, or routine), how challenging the situation was, how challenging the caller was, and how satisfied the caller appeared to be at the end of the call.
This live-monitoring protocol contains questions about call content and characteristics, suicide risk assessment, telephone counselor characteristics, call outcomes, and quality. Developed for use in a specific study of crisis call centers, the protocol may be useful for future evaluation efforts or for ongoing quality assurance purposes.
To download a PDF that includes the nine-page protocol, visit http://www.rand.org/pubs/tools/TL150.html.
Table 1. Review of Crisis Line Evaluation Studies
|Citation (listed chronologically)||Evaluation Design||Evaluation Findings|
|Weiner, 1969||A comparison of suicide rates in Los Angeles County before and after the introduction of a crisis hotline. Also, comparisons were made with the suicide rates in other California counties (one of the other three counties had a prevention program, two did not).||Researchers did not find a decrease in the suicide rate of Los Angeles Country after implementation of the program, but rather an increase. The suicide rate seemed to increase slightly with the rise in number of calls.|
|Bidwell, Bidwell, and Tsai, 1971||An evaluation of the demographic data records from a three-year period from September 1, 1966, to August 31, 1969, of crisis hotline calls were compared with data from those who had died by suicide. Names were compared to see whether the reported names of those who had committed suicide were found within the call logs of the help line.||The findings support the hypothesis that suicidal attempters and suicides constitute two epidemiological populations, albeit overlapping, and that the crisis intervention method of the suicide prevention programs can reach the first group but not the second. In other words, the demographics of the callers more closely resembled the attempters group, rather than the suicide completion group.|
|Lester, 1971||The census tract of 214 callers (of 626 possible) was identified and correlated with census tracts of local suicides for 1966–68.||Census tracts in Buffalo with one or more suicide in 1966–68 accounted for 86 percent of callers and 81.6 percent of the population.|
|Litman, 1976||Among a group of persons in contact with a crisis center, this study compared an experimental group that received outbound calls (Continuing Relationship Maintenance, or CRM), once per week for an average of 18 months per person with a control group.||No differences in completed suicides, suicide risk, or willingness to accept help. CRM group was less likely to live alone, had more improved personal relationships, better use of professional help, and less depression.|
|Leenaars and Lester, 1995||Pearson correlation between provincial suicide rates and (a) absolute number of crisis centers, (b) density of crisis centers per capita, and (c) density of crisis centers per area.||All correlations negative, though no statistical tests of significance were performed.|
|Mishara and Daigle, 1997||Trained observers listened to and coded calls in real time to ascertain the relative effectiveness of the volunteers' various intervention styles on the reduction of psychological distress of the callers. The volunteers' ability to encourage the caller to make a “no suicide contract” was also assessed.||An overall decrease in depressed mood was found from the beginning to the end of calls, but depression only decreased in 14 percent of calls and remained the same in 85 percent of calls. There was also a significant decrease in suicide urgency from the beginning to the end of the call (urgency decreased in 27 percent of calls), especially for non-chronic callers. Contracts were made in 68 percent of calls, more frequently with chronic callers. Calls were classified as “Rogerian style” or “directive style.” Those volunteers using Rogerian style had significantly more decreases in caller depression and more contracts.|
|Fiske and Arbore, 2000||The study measured depressive symptoms, hopelessness, and life satisfaction before and after clients received 1 year of services (including warmline with both inbound and outbound calls) from the agency.||A paired t-test revealed a significant reduction in hopelessness among the clients. There were no significant changes in depressive symptoms or life satisfaction. There were no changes in hopelessness, depressive symptoms, or life satisfaction in the comparison group.|
|King, Nurcombe, Bickman, Hides, and Reid, 2003||Independent raters quantify changes in suicidality over the course of a call or counseling session by reviewing the first 5 minutes when suicidality first became evident and last 5 minutes of the call.||Decreases in callers' mental state and suicidal ideation occurred from the beginning to the end of the call; a decrease in calls rated to be at “imminent risk” and an increase in those rated as “no suicide urgency” was also observed.|
|Mishara, Houle, and Lavoie, 2005||Pre-test, post-test, and follow-up questionnaires were administered to participants who received each of five different support styles, including telephone counseling, though participants were not randomly assigned. Questionnaires contained questions about the callers themselves as well as about the suicidal man. Questionnaires to family/friends addressed issues such as coping mechanisms and utilization of resources, whereas the questionnaires related to the suicidal man included topics such as suicidal behaviors and alcoholism. Some topic areas overlapped. No control group.||There were no differences across the five support styles. Participants reported that suicidal men were less likely to have suicide attempts or ideation and depressive symptoms post-training, and these effects were maintained at the 6-month follow-up. The programs did not increase knowledge/use of resources for the participants or suicidal man. Participants reported that treatment did not reduce the suicidal man's use of alcohol/drugs. On the pre-test questionnaire, participants also reported some reasons for not discussing the man's suicidal intentions with him: 32 percent cited not wanting to upset the suicidal person and 21 percent reported feeling embarrassed or ashamed to discuss the issue of suicide.|
|Mishara, Chagnon, and Daigle, 2007a||Trained observers listened to and coded calls in real time. The professional helpers were rated on different categories: their ability to conduct a suicide risk assessment in accordance with American Association of Suicidology accreditation, their ability to send emergency rescue if needed, and their ability to intervene according to existing theories related to active listening and collaborative problem-solving models.||81 percent of calls had a good initial rapport between helpers and callers. Only one-half of helpers asked about suicidal ideation. Of the callers who were reporting ideation, 46 percent were not asked about a plan; most were not asked about prior attempts.|
|Mishara, Chagnon, and Daigle, 2007b||Trained observers listened to and coded calls in real time. This evaluation is related to Mishara, Chagnon, and Daigle, 2007a. It looks to analyze whether there is a correlation between the behavior of the helpers and any short-term outcomes seen in the callers.||Empathy, respect, supportive approach, good contact, and collaborative problem solving were significantly related to positive outcomes. Active listening was not related to outcomes.|
|Meehan and Broom, 2007||Call logs were completed by volunteers, and 535 callers between March and September 2004 were mailed a questionnaire on their perceptions of the service (only 41 mailed the form back). The form included satisfaction for call, reasons for call, and time it took after learning about hotline to call.||Demographic data on callers presented; those who completed the questionnaire were generally happy with how their call was handled.|
|Gould, Kalafat, Munfakh, and Kleinman, 2007||Counselors at eight crisis centers conducted standardized assessments at the beginning and end of calls, and also asked if they could follow-up in 1–2 weeks with the caller. Follow-up calls were made by independent research interviewers.||Seriously suicidal individuals reached out to telephone crisis services. Significant decreases in suicidality were found during the course of the telephone session, with continuing decreases in hopelessness and psychological pain in the following weeks. A caller's intent to die at the end of the call was the most potent predictor of subsequent suicidality.|
|Kalafat, Gould, Munfakh, and Kleinman, 2007||Counselors at eight crisis centers conducted standardized assessments at the beginning and end of calls, and also asked if they could follow up in 1–2 weeks with the caller. Follow-up calls were made by independent research interviewers.||Significant decreases in callers' crisis states and hopelessness were found during the course of the telephone session, with continuing decreases in crisis states and hopelessness in the following weeks. A majority of callers were provided with referrals and/or plans of action for their concerns, and approximately one-third of those provided with mental health referrals had followed up with the referral by the time of the follow-up assessment. While crisis service staff coded these callers as nonsuicidal, at follow-up nearly 12 percent of them reported having suicidal thoughts either during or since their call to the center.|
|Ho, Chen, Ho, Lee, Chen, and Chou, 2011||The evaluation uses a pre-test/post-test design to evaluate the effectiveness of a center's programs, using monthly Bureau of Health data to track suicide rate changes since the center's opening in 2006.||From 2005 to 2008, suicide rates decreased, Kaohsiung Suicide Prevention Center (KSPC) crisis line calls increased, the number of KSPC telephone counseling sessions increased, and suicide attempt reporting increased.|
|Gould, Munfakh, Kleinman, and Lake, 2012||Lifeline callers who had received a mental or behavioral health care referral were interviewed two weeks after their call to assess depression, referral follow-through, and barriers to utilization both in suicidal callers and non-suicidal crisis callers.||Decreases in callers' mental state and suicidal ideation occurred from the beginning to the end of the call; a decrease in calls rated to be at “imminent risk” and an increase in those rated as “no suicide urgency” were also observed.|
|Knox, Kemp, McKeon, and Katz, 2012||Administrative data on calls to the Veteran's Crisis Line, which was established in July 2007, are reviewed.||Since the inception of the Department of Veterans Affairs' (VA) suicide hotline, the percentage of veterans self-identifying as veterans has increased from 30 percent to just over 60 percent, as of September 30, 2010; the volume of calls as of this time was 171,000. Seventy percent of callers were male veterans, and those who disclosed their age were between 40 and 69 years old. Approximately 4,000 referrals were made to the VA's suicide prevention coordinators as of 2008; there were 16,000 referrals at the end of September 2010.|
|Gould, Cross, Pisani, Munfakh, and Kleinman, 2013||Trained observers listened to and coded calls in real time across 17 call centers nationwide. Centers were offered staggered ASIST training, and analyses used hierarchical regression to evaluate relevant outcomes (counselors' interventions, callers' behavior change, the relation between the two, and effects over time) on the basis of whether centers had or had not received ASIST training.||Call counselors trained in ASIST had significantly positive intervention behaviors on six of 23 metrics, including longer calls and four of seven behavior changes (less suicidal, depressed, and overwhelmed; more hopeful). No relationship between time since training and outcomes. All behaviors that ASIST significantly impacted were associated with improved caller outcomes.|
Table 2. Interrater Reliability for N = 18 Calls
|Gender||Percent Agreement: 100|
|Call content||Percent Agreement: 61–94|
|Current suicidal ideation||Percent Agreement: 80|
|Past suicidal ideation||Percent Agreement: 89|
|Past suicide attempt||Percent Agreement: 94|
|Counselor response||Intraclass correlations (ICCs): Discussed feelings = 0.00, Reflected back feelings = 0.53, Reflected back situation = 0.37, Connected with caller = 0.60, Was sensitive/receptive = 0.58, Was respectful = 0.37, Showed empathy/validation = 0.56|
|Caller distress—beginning||ICC: 0.92|
|Caller distress—end||ICC: 0.79|
|Overall rating: rapport||Percent Agreement: 50, or ICC = 0.49|
|Overall rating: collaborative problem solving||Percent Agreement: 72, or ICC = 0.46|
|Overall rating: referrals||Percent Agreement: 94, or ICC = 0.89|
|Challenging situation||ICC: 0.44|
|Challenging caller||ICC: 0.78|
|Caller satisfaction||ICC: 0.65|
Gould, M. S., W. Cross, A. R. Pisani, J. L. Munfakh, and M. Kleinman, “Impact of Applied Suicide Intervention Skills Training on the National Suicide Prevention Lifeline,” Suicide and Life-Threatening Behavior, Vol. 43, No. 6, 2013, pp. 676–691.
Gould, M. S., J. Kalafat, J. L. Munfakh, and M. Kleinman, “An Evaluation of Crisis Hotline Outcomes, Part 2: Suicidal Callers,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, 2007, pp. 338–352.
Gould, M. S., J. L. Munfakh, M. Kleinman, and A. M. Lake, “National Suicide Prevention Lifeline: Enhancing Mental Health Care for Suicidal Individuals and Other People in Crisis,” Suicide and Life-Threatening Behavior, Vol. 42, No. 1, 2012, pp. 22–35.
Ho, W. W., W. J. Chen, C. K. Ho, M. B. Lee, C. C. Chen, and F. H. Chou, “Evaluation of the Suicide Prevention Program in Kaohsiung City, Taiwan, Using the CIPP Evaluation Model,” Community Mental Health Journal, Vol. 47, No. 5, 2011, pp. 542–550.
Joiner, T., J. Kalafat, J. Draper, H. Stokes, M. Knudson, A. L. Berman, and R. McKeon, “Establishing Standards for the Assessment of Suicide Risk Among Callers to the National Suicide Prevention Lifeline,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, 2007, pp. 353–365.
Kalafat, J., M. S. Gould, J. L. Munfakh, and M. Kleinman, “An Evaluation of Crisis Hotline Outcomes, Part 1: Nonsuicidal Crisis Callers,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, 2007, pp. 322–337.
King, R., B. Nurcombe, L. Bickman, L. Hides, and W. Reid, “Telephone Counselling for Adolescent Suicide Prevention: Changes in Suicidality and Mental State from Beginning to End of a Counselling Session,” Suicide and Life-Threatening Behavior, Vol. 33, No. 4, 2003, pp. 400–411.
Mishara, B. L., and M. S. Daigle, “Effects of Different Telephone Intervention Styles with Suicidal Callers at Two Suicide Prevention Centers: An Empirical Investigation,” American Journal of Community Psychology, Vol. 25, No. 6, 1997, pp. 861–885.
Mishara, B. L., J. Houle, and B. Lavoie, “Comparison of the Effects of Four Suicide Prevention Programs for Family and Friends of High-Risk Suicidal Men Who Do Not Seek Help Themselves,” Suicide and Life-Threatening Behavior, Vol. 35, No. 3, 2005, pp. 329–342.
Mishara, B. L., F. Chagnon, and M. Daigle, “Comparing Models of Helper Behavior to Actual Practice in Telephone Crisis Intervention: A Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, 2007a, pp. 291–307.
Mishara, B. L., F. Chagnon, and M. Daigle, “Which Helper Behaviors and Intervention Styles Are Related to Better Short-Term Outcomes in Telephone Crisis Intervention? Results from a Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, 2007b, pp. 308–321.
National Suicide Prevention Lifeline, “Suicide Risk Assessment Standards,” April 17, 2007. As of November 10, 2014:
Weiner, I. W., “The Effectiveness of a Suicide Prevention Program,” Mental Hygiene, Vol. 53, No. 3, 1969, pp. 357–363.
 All call monitoring activities described in this document were reviewed by the RAND Human Subjects Protection Committee.
 See National Suicide Prevention Lifeline, “Suicide Risk Assessment Standards,” April 17, 2007.
 The Lifeline QI Monitoring Tool is designed for crisis hotlines that are part of the NSPL to “facilitate [s]upervision of counselors and guide the appraisal of their behaviors during calls.” It is available to members of the NSPL and was provided to us when we were developing our quality monitoring form.
This research was conducted in RAND Health, a division of the RAND Corporation.