B: Conceptual Model Guiding the Identification of Potential Indicators and Theoretical Basis for the Model

Overview

Through extensive engagement of subject-matter experts and in consultation with CDC, our team developed a conceptual model to guide the identification and selection of potential indicators that could be used to monitor BH in the PHE context using existing data sources.

PHEs are complex and overlapping. They interact with community conditions and chronic stressors to add to overall cumulative stress that affects communities, with disproportionate impacts on populations that have been historically marginalized (Luft, 2016; Klaiman et al., 2010; Lieberman-Cribbin et al., 2020; Davis et al., 2010). You can track BH impacts of PHEs, and the strengths and vulnerabilities that influence a community’s risk of impacts, through PH monitoring and surveillance of indicators across three key domains (Summary 1).

Upstream

Community strengths and vulnerabilities that influence the likelihood of behavioral health impacts

Midstream

Early signals of behavioral health impacts that could be mitigated by prompt action

Downstream

Later behavioral health impacts that show behavioral health is worsening, improving, or unchanged

As discussed in Section 2 of the toolkit, you can think of community strengths and vulnerabilities as upstream factors that influence the BH impacts of a PHE; early signals of BH impacts as midstream factors, as they indicate that the PHE has started to have negative consequences but there is still time to intervene and mitigate their effects; and later BH impacts as the downstream manifestations of BH impacts that, ideally, you could prevent with earlier PH action. For the proof-of-concept analyses included in this toolkit, we intentionally selected indicators from each of these domains to represent the continuum of BH impacts of PHEs.

Of note, many of the data sources we mention (e.g., social media data and web searches) could be useful for monitoring up-, mid-, or downstream indicators of BH impacts, depending on the indicators you select. For example, you might choose to monitor ED visits for suicidal ideation (i.e., a midstream indicator) and for suicide attempts (i.e., a downstream indicator), both of which could be informative.

Summary 2 displays these three domains, each with several subdomains. Some of these indicators are available to you in existing data, while others are more conceptual in nature. Because this is a conceptual framework, we include the latter indicators for completeness, but additional work may be needed to operationalize these indicators in your local context (e.g., rates of community members’ participation in well-being activities or volunteerism).

The Community Strengths and Vulnerabilities domain of this conceptual model contains four subdomains:

• BH supports and services
• social connections
• well-being
• unstable housing and employment.¹

Early Signals of BH Impacts contains three subdomains:

• coping
• signs of stress
• calls for help.

Finally, the Later BH Impacts domain consists of three subdomains:

• BH-associated mortality
• emergency care and crisis services
• resilience and growth.

Theoretical Basis for the Conceptual Model

Public Health Emergencies Can Have Significant and Lasting Impacts on Behavioral Health

PHEs—natural and manmade—pose health risks and, for some, cause BH issues that can persist long after infrastructure is repaired (Newnham et al., 2022). Given both the unpredictability and high potential for damage, you and your colleagues must be prepared to pivot and respond to the needs of your jurisdiction. The myriad BH impacts after a PHE are significant and vary over time. Therefore, bolstering individual, organizational, and community resilience is a key PH priority (Chandra et al., 2011; Karatsoreos and McEwen, 2011; Norris et al., 2008).

Commonly used figures, such as the “Emotional Highs and Lows After a Disaster” figure in Crisis Counseling Assistance and Training Program Guidance (Federal Emergency Management Agency and U.S. Department of Health and Human Services, 2016), can help you consider the big picture, but they must be adapted to the BH context (Summary 3). Your jurisdiction is often in various phases of response to several emergencies simultaneously, so event timelines that show a PHE as a discrete, stand-alone event are less directly applicable to BH impacts of PHEs.

At the individual level, people may experience a variety of symptoms, from general psychological distress to diagnosable mental health disorders, including acute stress disorder, complicated grief, depression, anxiety disorders, substance use, posttraumatic stress syndrome (PTSS), and posttraumatic stress disorder (PTSD). Research has shown that PHEs negatively affect people’s coping resources by disrupting social networks, routines, and gathering places (Acosta et al., 2018; Lowe et al., 2015; Norris et al., 2006).

There are emotional highs and lows associated with a PHE. Individuals and populations progress through several PHE phases: pre-disaster, impact, heroic, honeymoon, disillusionment, and reconstruction. Individuals and populations experience “emotional lows” at the time of the PHE impact, during the disillusionment phase, and often during anniversaries of the event. They experience emotional highs just after the PHE occurs, during the heroic and honeymoon phases, as well as when moving from disillusionment to reconstruction.

Source: Reproduced from Federal Emergency Management Agency and U.S. Department of Health and Human Services, 2016.

Depending on individual, organizational, and disaster-related factors and on available supports, BH symptoms may either resolve themselves or evolve into conditions that warrant intervention (Adams et al., 2019; Reijneveld et al., 2005). The leading predictor for severe long-term BH problems is an individual’s proximity to the event (Freedy, Resnick, and Kilpatrick, 1992; Goldmann and Galea, 2014). However, the magnitude of a PHE, an individual’s pre-PHE health conditions (Ayer et al., 2019), and the persistence of unmet needs post-PHE have also been shown to influence the development, severity, and resolution of post-PHE psychological issues (Boscarino, 2015; Hunt et al., 2018; Oe et al., 2016; Schwartz et al., 2015; Siskind et al., 2016).

Despite extensive literature on the BH impacts of PHEs, there are insufficient data—and thus, there is insufficient evidence—to help you predict, conclusively, how PHEs will affect individuals and specific subpopulations (e.g., adolescents in American Indian or Alaska Native communities; Black women) in your jurisdiction; how and under what instances comprehensive PH interventions implemented at the population level may mitigate the BH impacts for individuals with specific risk factors or specific subpopulations; or to what extent population-level interventions might shorten the duration of symptoms or modify the natural course of BH morbidity experienced by individuals and subpopulations. PH and BH agencies and practitioners do not have an established collaborative process to address BH during PHEs. In short, PH’s role in addressing BH prevention, mitigation, and recovery is evolving.

Public Health Emergencies Pose a Unique Challenge for Data Collection and Data Quality

In a PHE, data quality and availability may be compromised because of health care and other essential service disruption (e.g., power outages), population displacement, or damage to infrastructure (Clarke et al., 2018). Therefore, you could face challenges in selecting an appropriate sampling frame representative of the jurisdiction. Data collection from larger samples will likely contain individuals not exposed to the PHE, so your data will underestimate the burden of the outcome(s) of interest. Large numbers of people may evacuate your jurisdiction, making it difficult to know who was exposed and how intensely. Communication channels may be compromised, preventing outreach to exposed populations. Certain populations may not be represented in your data (e.g., children, people from racial/ethnic minority groups, people with undocumented status, people experiencing homelessness, people who speak languages other than English, people who are incarcerated). Furthermore, detecting psychological distress is complex, given the variability in severity and presentation. For children, it often requires clinical or parental input. In summary, BH surveillance is complicated by challenges associated with measurement error from underreporting or misclassification of the BH indicator of interest and with collecting high-quality data during and after PHEs.

Data Sources That Collect Behavioral Health Information in Routine Times Have Limitations

PH surveillance is the “ongoing, systematic collection, analysis, and interpretation of health-related data essential to planning, implementation, and evaluation of public health practice” (Thacker and Birkhead, 2008, p. 985). Even outside the PHE context, data sources that contain information about BH have limitations, and there are few examples of sources currently in use for surveillance purposes. We describe some of these examples and their respective limitations later.

Surveys

Data sources that capture BH conditions but with a significant time lag include surveys, such as the BRFSS and the related Youth Risk Behavior Surveillance System. These annual surveys collect rich information on health behaviors, and, although they can help you understand longitudinal trends, they are not timely enough to be useful in a rapidly evolving PHE. Reliable data may not be available at a sub-state level, and the surveys cannot be quickly revised to add questions that would allow you to examine emerging issues (Henning, 2004; Lyerla and Stroup, 2018). Specifically, BRFSS, a random-digit-dial nationally representative survey, releases new data every year. The survey is state-based, so your state’s survey may or may not include the optional modules (e.g., an anxiety and depression module). However, few states currently administer the BRFSS (Miller et al., 2017). These characteristics limit the timeliness and usefulness of these data in the PHE setting.

Data on ED and Hospital Encounters

The State Inpatient Databases, administered by the Healthcare Cost and Utilization Project and sponsored by the Agency for Healthcare Research and Quality (AHRQ), are potential data sources that you could use to examine trends in ED visits and inpatient admissions for psychiatric conditions (Owens et al., 2019), drug overdoses (Owens et al., 2014), encounters related to violence (Russo, Owens, and Hambrick, 2008), child maltreatment (Russo, Owens, and Hambrick, 2008), and other BH conditions. These encounter-level databases are available at the sub-state level, include all payers, and are compiled in a uniform format. You will need to purchase these databases and follow a data request process to obtain them from AHRQ. Of note, these databases capture only BH conditions that led to a hospital-based encounter.

Pharmacy Data

Pharmacy data (e.g., fills, sales, claims) and OTC medication sales data have the potential to be used for monitoring patterns at the population level that might suggest BH impacts of a PHE; pharmacy sales data have been used to detect prescriptions associated with influenza (Patwardhan and Bilkovski, 2012). Finally, data on dispensed prescriptions have recently been used to examine trends in drug prescribing during the COVID-19 pandemic, and you might use this source to monitor for signals of BH problems at the population level (Vaduganathan et al., 2020). Limitations of these data are that they lack such information as prescription indication (i.e., reason for using the medication) and new or refill prescription status (van Meijgaard et al., 2020).

Other Data Sources

There is very little literature on other data sources that are currently used for BH surveillance and therefore could be adapted to the PHE setting. Emerging federal efforts to strengthen syndromic surveillance for BH through NSSP are described in the overview of ED visit data. NSSP’s BioSense platform aggregates patient encounter data from EDs and (to varying degrees, depending on your jurisdiction) from urgent and ambulatory care centers, inpatient health care settings, and pharmacy and laboratory data. Other data streams (e.g., mortality data) are being considered for integration into BioSense.

These aggregated data create a timely picture of emerging health needs in your community through monitoring health care service use, which in turn can help you determine demands on the health care system and treatment capacity. One limitation of these sources is that applying syndromic surveillance to BH conditions can detect only acute mental health incidents that led to a patient encounter or a filled prescription, which provides an incomplete picture of the range of severity of BH impacts after a PHE (Pavlin et al., 2004). Furthermore, misclassification due to use and misuse of diagnosis codes in a patient’s electronic health record may affect the ability of syndrome surveillance definitions to accurately capture the true burden of ED visits for BH conditions.

Public Health Agencies Currently Use a Patchwork of Data Sources for Behavioral Health Surveillance in the Public Health Emergency Context

As of 2022, no dedicated surveillance systems exist to track BH risk or impacts in the PHE context. Instead of a comprehensive surveillance system, there exists only a patchwork of surveys and surveillance systems that each have limitations (CSTE, 2013). The fragmentation of the BH care system (e.g., consisting of providers for people with disabilities, substance use disorder treatment providers, inpatient and outpatient mental health services, EDs) is one contributor to this patchwork of data (Kaul and Sherman, 2017).

In the pre-COVID-19 era, measuring the impacts of PHEs on BH typically relied on rapid assessment procedures. Although not surveillance, these cross-sectional studies offer a potential alternative to data sources that may be collected only annually (Harris, Jerome, and Fawcett, 1997). Comprehensive data-collection procedures that employ surveys, focus groups, interviews, and other sources typically used in rapid assessments may not be feasible for your jurisdiction, particularly after a traumatic community event. Therefore, although rapid assessments serve an important purpose during and after PHEs, they do not involve ongoing, systematic data collection that allows you to monitor trends and detect early signals of population-level distress. The resource intensity and time-limited nature of rapid assessment procedures make them an important complement to the type of BH surveillance by public health officials that is the focus of this toolkit.

BH surveillance efforts may be informed by the guidelines used by the well-established surveillance systems, such as those for infectious disease. However, BH surveillance in the PHE context needs to include information on a broad array of outcomes, types of PHEs, community conditions, individual behaviors, thoughts (e.g., SI), and policy changes (e.g., alcohol pricing) that influence BH (Azofeifa et al., 2018). In addition, use of BH surveillance data needs to account for the fact that individual BH trajectories are nonlinear (for example, symptoms of PTSD can emerge months or even years after an event and may reemerge around the event anniversary) (Bountress et al., 2017; Karstoft et al., 2015). Gold-standard approaches for monitoring key BH indicators are limited. For example, SI is a key indicator, but it is difficult to measure; many more individuals contemplate suicide than the number who carry it out (Harmer et al., 2022).

A significant step forward in addressing some of these challenges was the CSTE’s development of 18 core indicators for substance use disorders and mental health surveillance by PH agencies, released in 2017 (CSTE, Surveillance Committee, 2017). After two rounds of pilot testing to improve the guidance, recommendations for an updated set of indicators (Version 3) were released in 2019 (Substance Use and Mental Health Indicators Subcommittee, 2019). This set includes such indicators as drug overdose mortality, alcohol-related crash deaths, serious adult mental illness in the past year, and ED visits for intentional self-harm. To our knowledge, as of 2022, the results of field testing these indicators in select geographies (although not in the PHE context) were not yet publicly available.

Data Triangulation and Integration May Offer Promising Advances Needed for Behavioral Health Surveillance

Given the drawbacks described, integrating multiple data sources with data available at the sub-state level can improve the accuracy of your prevalence estimates of BH conditions and symptoms. For example, integrating surveys with electronic health records can improve accuracy of estimates of depression prevalence (Davidson et al., 2018). As discussed in Part 4 of the toolkit, triangulating multiple data sources provides a more complete picture of (ideally) early signals of potential BH conditions at a population level. Integrating 2-1-1 call data with claims data from EDs for certain BH conditions could help you monitor evolving patterns of distress in your jurisdiction to inform PH interventions and collaborations with BH providers for ongoing and follow-up care. The Fusion Analytics program within the Administration for Strategic Preparedness and Response at the U.S. Department of Health and Human Services is one example of a web-based platform that integrates several data sources for the purpose of matching needs to available resources during a PHE. The platform displays information in easily interpretable data visualizations (Passman, 2013) (such as real-time patient encounter data and hospital bed availability), but dashboards are only accessible if you have an “hhs.gov” email address (Administration for Strategic Preparedness and Response, 2016). An important consideration is that data integration is resource-intensive and can require advanced data science capacity. Integration of data from multiple sources also requires thoughtfully addressing data governance issues, such as whether an integrated data repository will be created and, if so, who manages it and who can access and use those data.

Finally, as suggested by Stallings (1997), post-PHE health surveillance is most effective when using a mixed-methods approach. The prime advantage of several data collection methods—both qualitative and quantitative—is the triangulation or corroboration of findings, which can build evidence for prevention and response (Stallings, 1997). Given the subjectivity of BH conditions, it is often not straightforward to develop standard case definitions (the “who and what is counted” [Lyerla and Stroup, 2018]; therefore, a mixed-methods approach can be useful for validating certain types of data, such as those resulting from self-reports (Perou et al., 2013). Norris et al. (2006) recommend ongoing syndromic surveillance focusing on key indicators, such as depression, PTSS and PTSD, anxiety, and psychosocial resources. They also recommend that ongoing surveillance be punctuated by occasional, more-detailed disease-specific surveys that provide you with estimates of BH disorders along with a closer look at risk and protective factors (Norris et al., 2006). Their suggested indicators align with recent CSTE recommendations and emphasize the importance of mixed methods in capturing a fuller picture of BH impacts.

The conceptual model shown in Summary 2 is the result of considering the full variety of potential BH impacts of PHEs, developing an ideal list of indicators and data sources to capture those impacts, and recognizing that not all of the concepts listed can be captured in readily available, existing data sources.