C: Methods Used to Develop and Finalize the Toolkit

This toolkit was developed through a multistep process consisting of drafting and finalizing phases (Summary 1). We describe our methods to draft the toolkit first, followed by our methods for finalizing it. In each section, we provide a brief overview, then go into greater detail on our approach.

The methods to draft the toolkit include a literature review, an environmental scan, informational interviews with data producers, formative qualitative interviews, exploratory data analyses, and expert input. From these methods, we developed a list of the most-promising data sources to use for BH surveillance in PHEs and began drafting the toolkit. To finalize the toolkit, we conducted summative qualitative interviews and solicited additional expert input.

Methods Used to Draft the Toolkit

We used six methods to identify and describe possible data sources and the relevant indicators and analytic methods used with each data source. These methods were

• a literature review to summarize how PHEs affect BH and during what time frame(s); how PH agencies have been surveilling BH during PHEs, if at all; and the strengths and limitations of surveillance techniques that have been used to successfully identify early warning signs of future BH needs and problems

• an environmental scan to identify existing data sources that are available at the sub-state level and updated frequently enough to be useful in the PHE context, the extent to which those sources have been used to track and monitor BH, and whether they have been used in the PHE context

• informational interviews with data producers (e.g., the American Association of Poison Control Centers [AAPCC]) to better understand how PH agencies could access these data, the ways these data have been used for similar purposes, and their potential strengths and limitations

• formative qualitative interviews with officials from state, local, and tribal PH agencies and with subject-matter experts in PH and BH to solicit input on how BH surveillance data are used, or could be used, in the context of PHEs

• exploratory data analyses to assess a data source’s strengths and limitations for use in BH surveillance

• a technical expert panel meeting to solicit input from experts representing national, state, and local PH and mental health organizations and through ongoing interactions with experts from SAMHSA and CDC.

  Next, we describe each of these methods in more detail.

Literature Review

We conducted a targeted literature review to qualitatively summarize and assess available evidence on BH surveillance in the PHE setting by answering the following questions:

• How is BH affected by PHEs? What types of symptoms appear and during what time frame(s)?
• How have PH agencies been surveilling BH during PHEs?
• What types of surveillance techniques have been used to successfully identify early warning signs of future BH needs and problems? What are the strengths and limitations of these techniques?

We conducted the targeted literature search in four stages:

• identify potential sources using a keyword search
• review titles and abstracts to identify relevant sources
• abstract information from relevant sources by conducting a full-text review
• conduct a supplemental scan for relevant grey literature.

To identify sources, we conducted database searches of PubMed, PsycINFO, and Web of Science, using the following keywords: behavioral health, mental health, substance use, data sources, data needs, measurement, surveillance methodology, disaster, emergency, and public health management (Table 1). We limited the search to English-only systematic reviews and meta-analyses from 2014 to November 2019 (the date the search was conducted). We limited the search to only the past five years to capture the most-relevant and most-up-to-date literature. In addition, we reviewed reference lists for those sources that underwent full-text review to identify further sources. If particular authors appeared repeatedly during searches, we also searched by author and key terms. The initial search yielded 138 sources; 89 were excluded based on title and abstract review because they did not provide information on surveillance, did not discuss BH surveillance, or did not provide concrete strategies for surveillance during PHEs. After this step, 49 sources were left for full-text review. After excluding sources that did not warrant citation in this summary, adding sources identified from bibliographies of those undergoing full-text review, and including selected documents from the grey literature, we examined a total of 54 sources.

The literature review revealed myriad BH impacts that occur at different points during disaster response and recovery but no dedicated surveillance systems for tracking these impacts. Disaster experiences can result in psychological distress, acute stress disorder, complicated grief, depression, anxiety, and substance use disorders, among other impacts. Surveillance systems for infectious diseases, chronic diseases, and environmental PH are well established, but BH surveillance has thus far been limited. BH impacts are challenging to measure, especially if symptoms do not necessitate a visit to a health care provider. Disasters present additional challenges for surveillance because of population displacement, disruption in health care and other essential services, and infrastructure damage. Because no dedicated surveillance systems exist, PH leaders rely on a patchwork of existing surveys and surveillance systems created for other purposes. Although literature on promising practices for BH surveillance in a PHE is limited, what does exist suggests that active, passive, and syndromic surveillance techniques will be required to adequately measure BH impacts during and after PHEs. More research is needed on solutions to surveillance challenges in the PHE context.

Environmental Scan

For the environmental scan, we conducted a second literature search with the goal of identifying data sources that had been used to conduct BH surveillance, particularly those that had been used in the PHE context. Specifically, we searched for peer-reviewed publications in English during the time frame of 2010 to 2020 (the year the search was conducted) using two databases, PubMed and PsycINFO, and four groups of search terms related to PHE, BH, data, and data sources (Table 2). We opted to include articles from the prior ten years (rather than five for the more general literature review) to cast a slightly wider net for literature relevant to this more focused question. Titles and abstracts were reviewed to determine whether articles were about efforts to surveil, monitor, or track BH during a PHE. Of the 179 articles identified through the original search, 68 articles were included after title and abstract review. We also conducted a series of additional targeted scans of specific data sources using search terms tailored specifically for each data source (Table 2). Using inclusion criteria similar to the first literature search described in the previous section, we used Google Scholar to search for publications in English during the time frame of 2010 to 2020 that used existing data sources to monitor BH. We excluded articles describing studies that focused on primary data collection efforts (e.g., surveys or interviews) rather than secondary analyses of existing data sources. We then conducted a title and abstract review of the first five pages of search results, which yielded an additional 66 unique articles, 58 of which were retained after a title and abstract review.

We also systematically reviewed the reference lists of articles identified in the initial literature review to identify any articles that met our inclusion criteria (e.g., printed in English, published in the past ten years, used an existing data source to monitor BH). We included 29 additional articles identified through this scan of publication titles. Two of these articles were excluded after an abstract scan, resulting in 27 being included for review.

During data abstraction, the full text of articles was reviewed, which resulted in 18 additional articles being excluded, primarily because they contained only references to methods and indicators that would require new data collection rather than relying on existing data sources. We developed a structured data abstraction form to collect consistent information from each of the 135 peer-reviewed articles. For example, we documented whether the study was conducted in the PHE setting (and, if so, which type of PHE), the time frame of the study, the indicator(s) of interest, the lowest level of spatio-temporal granularity, the analytic methods used, and key strengths and limitations of the analysis. Three researchers on our team abstracted the data, and two senior-level researchers and a statistician spot-checked the abstracted data for accuracy. In total, we abstracted data from 135 articles, representing 11 unique types of data sources under consideration (or currently being used) for BH surveillance during PHEs. Most of the data sources had at least three articles that described their use and application, while ED records, web searches, social media, EMS, and pharmacy data were the most commonly used data sources for monitoring BH during a PHE and related purposes. Five articles reported on such miscellaneous data sources as local electronic health records or police records for a particular city, and two articles covered multiple data sources.

Informational Interviews with Data Producers

To supplement the information gathered from the literature search, we also held informational interviews with data producers (e.g., the AAPCC, NEMSIS, the NRDM) and reviewed relevant websites describing the data sources of interest to better understand how PH agencies could access these data, the ways that these data have been used for similar purposes, and their potential strengths and limitations.

Formative Qualitative Interviews

To solicit input on how BH surveillance data are used or could be used in the context of PHEs, we conducted 11 formative interviews with state and local PH agencies and subject-matter experts in PH and BH. These individuals were selected for their deep knowledge of available data sources and their limitations. These formative interviews were primarily used to help identify promising approaches and candidate data sources for the set of exploratory analyses. The stakeholders were selected collaboratively between our research team and CDC, and they had the following areas of expertise:

• two epidemiologists with expertise in BH or disaster BH
• two clinical psychologists with expertise in disaster BH and trauma
• two emergency preparedness and response professionals who worked in disaster BH response and recovery
• two physicians with expertise in disaster epidemiology, emergency preparedness and response, mental illness, and/or substance use; both had experience working in state PH or mental health departments
• the director of a state BH division
• a PH and BH strategist involved in the state’s BH surveillance efforts
• a subject-matter expert in mental health burden, mental health care use and costs, and intervention opportunities.

Over the course of the 60-minute conversations, conducted by videoconference, we used a semistructured discussion guide that included questions about the need for BH surveillance in PHEs, challenges and trade-offs with BH surveillance, promising indicators and data sources that could be used for BH surveillance, potential uses of BH information, and recommendations for the toolkit. Members of our team with expertise in qualitative methods conducted all interviews, which were audio-recorded and professionally transcribed. The interview transcripts were coded and ana­lyzed for common themes focused on each discussion question. We developed a coding scheme that reflected the discussion guide and used the codebook to train coders. Two team members coded each transcript in Dedoose, a qualitative data management software program. When there were discrepant views, coders discussed discrepancies until they were resolved. Discussions involved the interviewer when necessary.

Exploratory Data Analyses

To explore whether several of the most-promising data sources provided useful information associated with a PHE, we conducted five sets of exploratory analyses using data sources that we deemed promising for BH surveillance in PHEs and that were accessible to our team as non-PH officials. We also compiled real-world examples of the use of two data sources for this purpose (ED visits and EMS activations). The data sources selected for more exploration were

• UI claims
• 2-1-1 calls
• PCC calls
• prescription medication fills
• OTC sleep aid sales.

The methods and findings of these analyses are described elsewhere in the toolkit.

Expert Input

In October 2020, we convened a two-hour meeting by videoconference with ten invited technical experts to participate in a structured discussion of the following questions:

• What is the current state of BH surveillance in PHEs?
• What are the opportunities to strengthen BH surveillance in PHEs?
• How can the gap be closed between the current state of BH surveillance and the more ideal state?

The technical expert panelists’ insights directly informed next steps of this work.

Most-Promising Data Sources

We then rated each data source using the data collected using the aforementioned six methods. We looked for three attributes:

• The data are available to PH agencies at the sub-state level (e.g., city, county).
• The data are made available to data users more frequently than once a year.
• The data are available in most jurisdictions in the United States.

These attributes were deemed critical for the data source to be of use to a PH or mental health department seeking to understand, in a timely way, the rapidly changing population-level BH impacts of a PHE in its jurisdiction.

We assessed 27 unique data sources that contained indicators relevant to BH. Eight met the three criteria and were deemed appropriate for further exploration and inclusion in this toolkit. (We refer to these eight as the most-promising data sources.) Section 2 of the toolkit contains the qualitative ratings for these eight data sources. From that part of the toolkit, you can navigate to the exploratory analyses and/or real-world examples of the use of seven of these data sources (the eighth, the 9-8-8 suicide and crisis lifeline, went live as this toolkit was going into production, so we were not able to obtain and analyze data from this data source). Table 3 displays qualitative ratings of the other 19 data sources that we considered but deemed less promising for BH surveillance in the PHE context. Green means that the selected attribute was most optimal for that data source; red means that it was least optimal, and yellow indicates that it was mixed for that data source.

Methods Used to Finalize the Toolkit

Once a draft of the toolkit was completed, we used two additional methods to finalize it:

• summative qualitative interviews with state, tribal, and local PH representatives; subject-matter experts in PH and BH; and data source experts to solicit input on (1) how our findings could be interpreted and acted on and (2) the big picture of current and future opportunities for BH surveillance in PHEs
• a second meeting of the technical expert panel of national, state, and local PH and mental health stakeholders, complemented by ongoing interactions with experts from SAMHSA and CDC.

Summative Qualitative Interviews

To solicit input on how BH surveillance in the context of PHEs is carried out and on the exploratory data analyses, we conducted summative interviews with state, tribal, and local PH representatives; subject-matter experts in PH and BH; and data source experts.

Thirty-seven summative interviews were conducted by videoconference with a total of 43 individuals, selected collaboratively by our team and CDC. The final sample had the following areas of expertise:

• four state or local PH and emergency management officials
• nine state or local BH officials
• 12 federal BH experts
• 15 experts on specific data sources
• three experts on BH surveillance or surveillance during PHEs.

Over the course of the 60-minute conversations, we used a semistructured discussion guide that included questions about BH surveillance activities, feedback on the exploratory data analyses, considerations around how best to strengthen BH surveillance, and insights on other data sources and promising efforts with which to align. The discussion guide was developed by our team; members of our team with expertise in qualitative methods conducted all interviews, which were audio-recorded and professionally transcribed. The interview transcripts were coded, and coded segments were analyzed for common themes focused on data sources and other discussion questions. Two of our team members coded the transcripts in Dedoose. When there were discrepant views, coders discussed discrepancies until they were resolved.

Technical Expert Input

Two years after their initial meeting, in October 2022, we again engaged the technical expert panel members to review the draft of this toolkit and provide either written comments using a structured feedback form or verbal comments during a virtual meeting with our team.

To finalize the toolkit, we incorporated the feedback from the technical experts and the findings from the summative qualitative interviews, as well as input from multiple reviewers at CDC and SAMHSA.