Skip to main content of tool

Monitoring and Surveillance of Behavioral Health in the Context of Public Health Emergencies

A Toolkit for Public Health Officials

Contents

G: Publications Illustrating the Use of Selected Data Sources

Selected Publications Linking Employment Data with Behavioral Health or Public Health Emergencies

Study Objectives and Analytic Approaches

  • We identified five articles describing trends over time using unemployment rates and annual gross wages or income across a state, region, or county. Between five and fifteen years of data were used to establish trends. Authors used methods ranging from simple correlations to more-sophisticated modeling for autocorrelated longitudinal data to do the following:
    • Examine whether admissions to the ED for mental health–related conditions are associated with state-level monthly unemployment rates over time using time series analyses, as well as ARIMA analyses to determine whether lag times between changes in unemployment rates and ED admissions were significant (Bidargaddi et al., 2015).
    • Explore the impact of a large fire at a fireworks factory on wages of residents in the affected areas by using fixed effects panel regression to compare wages overall and among a low-wage group between a matched random sample of residents in affected vs. non-affected areas (van der Velden et al., 2019).
    • Explore the impact of a hurricane (Hurricane Ike, Texas 2008) on county unemployment by using a spatial Durbin panel model to quantify the relationship between employment and number of businesses in a county and then estimating the impact of the hurricane on employment using the number of open businesses in the three months after the hurricane (LeSage et al., 2011).
  • Two studies aimed to detect the magnitude and persistence of the effects of a variety of shocks on employment.
    • The first used a structural break-point analysis to estimate the number, timing, and confidence intervals of structural breaks in the time series of unemployment rate differences for four case-control samples after major Midwest flooding: small lightly damaged counties, large lightly damaged counties, small heavily damaged counties, and large heavily damaged counties (Xiao and Feser, 2014).
    • The second used ordinary least squares regression to examine differences in unemployment and income in counties and parishes by the extent of their damage from Hurricane Katrina (categorizing locations most affected as the “core” and those that share boundaries with the core as the “edge”) (Xiao and Nilawar, 2013).

Key Findings

  • Several of these studies found a relationship between a shock (e.g., a PHE such as a hurricane or flood) and unemployment rates.
  • For example, one study estimated that Hurricane Ike resulted in 274,000 lost jobs in Harris County and predicted that a loss of 3,000 jobs would persist up to three years after the hurricane (LeSage et al., 2011). Similarly, another study found that the impact of a major fire on unemployment and wages persisted three years after the disaster (van der Velden et al., 2019). After Hurricane Katrina, “core” areas affected by the hurricane in Alabama, Mississippi, and Louisiana became “doughnut holes” of low income and employment growth, while counties and parishes along the “edge” experienced high employment growth (Xiao and Nilawar, 2013). In contrast, studies of the impact of flooding on unemployment showed only short-term impacts, suggesting that unemployment impacts vary by disaster type (Xiao and Feser, 2014).
  • Other articles establish a relationship between unemployment and mental health, which helps connect the dots between a PHE and downstream mental health impacts. Time series modeling found that between one-third and two-thirds of the variation in ED admissions for mental health–related conditions could be predicted by the unemployment rates from the prior month (Bidargaddi et al., 2015).

Selected Publications Linking 2-1-1 Calls with Behavioral Health or Public Health Emergencies

  • Sherry I. Bame, Kay Parker, Jee Young Lee, Alexandria Norman, Dayna Finley, Atmaja Desai, Abha Grover, Courtney Payne, Andrew Garza, Ashley Shaw, et al., “Monitoring Unmet Needs: Using 2-1-1 During Natural Disasters,” American Journal of Preventive Medicine, Vol. 43, No. 6, Supp. 5, December 2012. https://pubmed.ncbi.nlm.nih.gov/23157762/
  • Katherine S. Eddens, Kassandra I. Alcaraz, Matthew W. Kreuter, Suchitra Rath, and Regina Greer, “A 2-1-1 Research Collaboration: Participant Accrual and Service Quality Indicators,” American Journal of Preventive Medicine, Vol. 43, No. 6, December 1, 2012. https://www.sciencedirect.com/science/article/abs/pii/S0749379712006290
  • Nasser Sharareh, Rachel Hess, Neng Wan, Cathleen D. Zick, and Andrea S. Wallace, “Incorporation of Information-Seeking Behavior into Food Insecurity Research,” American Journal of Preventive Medicine, Vol. 58, No. 6, June 1, 2020. https://www.sciencedirect.com/science/article/abs/pii/S0749379720300441

Study Objectives and Analytic Approaches

  • Among the three studies identified as using 2-1-1 call data for purposes that are relevant to this toolkit, indicators of interest were the proportion of daily calls for a particular concern, call volume per day, and rate of calls (number of calls in a particular area divided by population size). Authors used the following methods:
    • Descriptive univariate analysis to describe the proportion of total calls that were related to health or safety concerns and the median and range of daily call volume in relation to heat waves and other events (Eddens et al., 2012)
    • Simple inference using regression and t-tests to examine the association of call volume with research participant accrual and service quality indicators
    • Spatial mapping and analyses (e.g., optimized outlier analysis) to identify hot spots of unmet disaster-related needs among evacuees of Hurricanes Katrina and Rita (Bame et al., 2012) and hot spots of food insecurity (Sharareh et al., 2020). For instance, Sharareh et al. (2020) used multinomial logistic regression to identify factors associated with information-seeking about food resources and resource use rates to identify areas with mismatched information needs and available resources. This approach could be useful for PH agencies interested in spatial analyses of the mismatch between supply and demand for BH services.

Key Findings

  • In one study of more than 635,000 calls to 2-1-1 centers in Texas, nearly one in five calls were related to health and safety, although the authors did not present more-detailed information about BH needs versus PH ones (Bame et al., 2012). The authors found that unmet need “surged” during evacuation and immediate disaster response but remained elevated through the recovery period, which they defined as the three months after Hurricane Rita. They were able to show that, adjusting for population size, hot spots of unmet need appeared in smaller evacuation destinations (presumably because of strained local resources) and along evacuation routes.
  • Another study of 2-1-1 calls over an 18-month period in Missouri found that call volume periodically spiked to more than 700 calls per day during times of crisis for Missourians (Eddens et al., 2012). Call volume ranged widely, from 26 to 2,390 calls per day, with a median of 554 during the study period. Call volume spiked (a spike was defined as an increase in volume over a threshold set by 2-1-1 Missouri of 700 calls per day) during two heat waves, one in 2010 and one in 2011.
  • In these studies, the lowest geospatial levels examined were states in one study, counties in another, and zip codes aggregated to Small Health Statistics Areas in the third.

Selected Publications Linking Calls to Poison Control Centers with Behavioral Health or Public Health Emergencies

  • Joseph E. Carpenter, Arthur S. Chang, Alvin C. Bronstein, Richard G. Thomas, and Royal K. Law, “Identifying Incidents of Public Health Significance Using the National Poison Data System, 2013–2018,” American Journal of Public Health, Vol. 110, No. 10, October 2020. https://www.ncbi.nlm.nih.gov/pubmed/32816555
  • Alvin F. Chu, Steven M. Marcus, and Bruce Ruck, “Poison Control Centers’ Role in Glow Product–Related Outbreak Detection: Implications for Comprehensive Surveillance System,” Prehospital and Disaster Medicine, Vol. 24, No. 1, January–February 2009. https://pubmed.ncbi.nlm.nih.gov/19557960/
  • Kelly R. Klein, Perri Herzog, Susan Smolinske, and Suzanne R. White, “Demand for Poison Control Center Services ‘Surged’ During the 2003 Blackout,” Clinical Toxicology, Vol. 45, No. 3, 2007. https://www.ncbi.nlm.nih.gov/pubmed/17453875
  • R. K. Law, S. Sheikh, A. Bronstein, R. Thomas, H. A. Spiller, and J. G. Schier, “Incidents of Potential Public Health Significance Identified Using National Surveillance of US Poison Center Data (2008–2012),” Clinical Toxicology, Vol. 52, No. 9, November 2014. https://www.ncbi.nlm.nih.gov/pubmed/25175899
  • A. R. Nathan, K. R. Olson, G. W. Everson, T. E. Kearney, and P. D. Blanc, “Effects of a Major Earthquake on Calls to Regional Poison Control Centers,” Western Journal of Medicine, Vol. 156, No. 3, March 1992. https://www.ncbi.nlm.nih.gov/pubmed/1595244
  • Amy F. Wolkin, Colleen A. Martin, Royal K. Law, Josh G. Schier, and Alvin C. Bronstein, “Using Poison Center Data for National Public Health Surveillance for Chemical and Poison Exposure and Associated Illness,” Annals of Emergency Medicine, Vol. 59, No. 1, January 2012. https://pubmed.ncbi.nlm.nih.gov/21937144/

Study Objectives and Analytic Approaches

  • Across the six studies providing relevant information about the potential use of PCC calls to monitor BH impacts of a PHE, the two most common indicators were daily call volume and number of call anomalies within the study period (defined as an hourly call volume more than three standard deviations above a historical baseline).
  • Studies looked at time trends and used statistical detection algorithms to detect an aberration from the trend or conducted change analysis before and after an event.
  • For example, one study by Nathan et al. (1992) used descriptive statistics (frequencies and proportions) to compare the frequencies and proportions of calls before and after a large earthquake, describing differences in the demographics of callers, reason for exposure, route of exposure, and medical outcome. Of note, this study was one of the only analyses of PCC data that reported on the frequency and proportion of calls related to suicide. Another, by Klein et al. (2007), reported the change in daily calls to a PCC after a widespread power outage in 2003, comparing August 2002 with August 2003. Another study simply reported trends in number of calls related to a particular product type (“glow products”) throughout the year compared with on the day of Halloween of that year, and on the day of Halloween in different years (Chu, Marcus, and Ruck, 2009). Several studies (Carpenter et al., 2020; Law et al., 2014; Wolkin et al., 2012) examined the use of PCC data for detecting “events of public health significance” due to a mass exposure, which is outside the scope of our project and its focus on BH.
  • Of note, these studies typically reported changes in call volume immediately after an event (such as a large earthquake or widespread power outage). For BH impacts, one would expect changes in call volumes related to intentional exposures to occur with more of a delay.

Key Findings

  • One study found that in the immediate 12 hours after an earthquake, there was an initial drop of 31 percent in PCC call volume due to telephone system overload and other technical problems (Nathan et al., 1992).
  • Another found that the average call volume for August 2003 (the month after a widespread power outage occurred in New Jersey) increased by 7.8 percent compared with August 2002 (Klein et al., 2007). Compared with the same time and day in the previous week, the total number of calls received by the New Jersey PCC during the four hours after the power outage increased by 148 percent. Again, these studies are examining the immediate aftermath of these PHEs, while our project would require a much longer time frame of observation to identify intentional ingestions or exposures.

Selected Publications Linking Prescription Medication Fill Data with Behavioral Health or Public Health Emergencies

  • Sarah Axeen, “Trends in Opioid Use and Prescribing in Medicare, 2006–2012,” Health Services Research, Vol. 53, No. 5, October 2018. https://www.ncbi.nlm.nih.gov/pubmed/29532477
  • Jian-Hua Chen, K. Schmit, H. Chang, E. Herlihy, J. Miller, and P. Smith, “Use of Medicaid Prescription Data for Syndromic Surveillance—New York,” Morbidity and Mortality Weekly Report, Vol. 54, Supplement, August 26, 2005. https://pubmed.ncbi.nlm.nih.gov/16177690/
  • Charles DiMaggio, Sandro Galea, and Paula A. Madrid, “Population Psychiatric Medication Prescription Rates Following a Terrorist Attack,” Prehospital and Disaster Medicine, Vol. 22, No. 6, November–December 2007. https://pubmed.ncbi.nlm.nih.gov/18709935/
  • T. Dorn, C. J. Yzermans, and J. van der Zee, “Prospective Cohort Study into Post-Disaster Benzodiazepine Use Demonstrated Only Short-Term Increase,” Journal of Clinical Epidemiology, Vol. 60, No. 8, August 2007. https://www.ncbi.nlm.nih.gov/pubmed/17606175
  • Thijs Fassaert, Tina Dorn, Peter M. M. Spreeuwenberg, Martien C. J. M. van Dongen, Christel J. A. W. van Gool, and C. Joris Yzermans, “Prescription of Benzodiazepines in General Practice in the Context of a Man-Made Disaster: A Longitudinal Study,” European Journal of Public Health, Vol. 17, No. 6, December 2007. https://pubmed.ncbi.nlm.nih.gov/17412715/
  • Kyu-Man Han, Kyoung-Hoon Kim, Mikyung Lee, Sang-Min Lee, Young-Hoon Ko, and Jong-Woo Paik, “Increase in the Prescription Rate of Antidepressants After the Sewol Ferry Disaster in Ansan, South Korea,” Journal of Affective Disorders, Vol. 219, September 2017. https://www.ncbi.nlm.nih.gov/pubmed/28505500
  • Jonathan S. Levin, Joie Acosta, and Laura J. Faherty, “New Prescription Fills of Selective Serotonin Reuptake Inhibitors Before and During the COVID-19 Pandemic in Los Angeles County, California,” Journal of Affective Disorders, August 30, 2022. https://pubmed.ncbi.nlm.nih.gov/36055533/

Study Objectives and Analytic Approaches

  • Several relevant studies in the peer-reviewed literature used pharmacy data to examine prescription rates of psychotropic medications (e.g., antidepressants, benzodiazepines) after a disaster.
  • We also include a study of trends in opioid use and prescribing in Medicare over a seven-year period to explore its potential utility for the purposes of our project (Axeen, 2018), primarily to demonstrate the clinical details that are available in this data source, including diagnoses associated with prescriptions provided, duration, quantity, and suspected misuse. We identified a small number of studies from outside the United States (Dorn, Yzermans, and van der Zee, 2007; Fassaert et al., 2007; Han et al., 2017) whose methods (although not the data sources themselves) are highly relevant for our purposes.
  • Finally, we included a publication on the use of Medicaid prescription data for syndromic surveillance (Chen et al., 2005); again, it contained useful information about this data source and potential analytic methods we are considering for our work.
  • One study assessed the agreement between prescription medication data and diagnoses in outpatient records reporting sensitivity and specificity statistics. The other studies primarily used regression analysis (e.g., multivariable logistic regression, as well as regression-based quasi-experimental approaches, such as DiD analyses and ITSAs) to examine changes in prescription rates of certain medications (e.g., benzodiazepines, antidepressants) over time.

Key Findings

  • The key findings across these studies demonstrate the nuanced information that prescription medication fill data, particularly Medicaid claims, can provide.
  • For example, one study showed that disaster victims were at an increased risk of “becoming an incident benzodiazepine user” after the disaster but that prolonged use was not often observed and did not differ between disaster victims compared with references (Dorn, Yzermans, and van der Zee, 2007). Similarly, after a large fire, affected patients were not more likely to use benzodiazepines, but parents of disaster victims were (in the short term only) (Fassaert et al., 2007).
  • A study of the prescription rate of antidepressants after a South Korean ferry disaster found a significant increase (5.6 percent) in the prescription rate of antidepressants but not anxiolytics or sedatives and/or hypnotics (using a centralized South Korean database, the Health Insurance Review and Assessment Service database [Han et al., 2017]). It should be noted that this study was included for its relevant methods rather than for the data source itself.
  • Additionally, a study using Medicaid analytic extract data from New York residents found that, from September to December 2001, among individuals residing within three miles of the World Trade Center site, there was a statistically significant 18.2-percent increase in the SSRI prescription rate compared with the previous eight-month period, but a 9.3-percent increase for residents outside New York City was not statistically significant (DiMaggio, Galea, and Madrid, 2007).

Selected Publications Linking Over-the-Counter Sleep Aid Sales with Behavioral Health or Public Health Emergencies

  • Debjani Das, K. Metzger, R. Heffernan, S. Balter, D. Weiss, and F. Mostashari, “Monitoring Over-the-Counter Medication Sales for Early Detection of Disease Outbreaks—New York City,” Morbidity and Mortality Weekly Report, Vol. 54, Supplement, August 26, 2005. https://www.ncbi.nlm.nih.gov/pubmed/16177692
  • Anna Goldenberg, Galit Shmueli, Richard A. Caruana, and Stephen E. Fienberg, “Early Statistical Detection of Anthrax Outbreaks by Tracking Over-the-Counter Medication Sales,” Proceedings of the National Academy of Sciences of the United States of America, Vol. 99, No. 8, April 16, 2002. https://www.ncbi.nlm.nih.gov/pubmed/11959973
  • William R. Hogan, Fu-Chiang Tsui, Oleg Ivanov, Per H. Gesteland, Shaun Grannis, J. Marc Overhage, J. Michael Robinson, and Michael M. Wagner, “Detection of Pediatric Respiratory and Diarrheal Outbreaks from Sales of Over-the-Counter Electrolyte Products,” Journal of the American Medical Informatics Association, Vol. 10, No. 6, November–December 2003. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC264433/
  • Stephen F. Magruder, S. Happel Lewis, A. Najmi, and E. Florio, “Progress in Understanding and Using Over-the-Counter Pharmaceuticals for Syndromic Surveillance,” Morbidity and Mortality Weekly Report, Vol. 53, Supplement, September 24, 2004. https://www.ncbi.nlm.nih.gov/pubmed/15714640

Study Objectives and Analytic Approaches

  • We identified very few relevant studies using OTC medication sales as a timely source of data for surveillance purposes.
  • One article (Goldenberg et al., 2002) reviews the advantages of using “grocery and OTC” medication sales for detection of an outbreak, and we opted to include it in our environmental scan because of its detailed information on the timeliness and other attributes of this data source, as well as its description of a “modular detection system, composed of several layers, where each layer applies a statistical tool to an OTC sales series” to detect a specific “disease footprint.” The statistical tools are described in detail, including steps to process the data and eliminate noise.
  • Another study (Das et al., 2005) examined the daily ratio of sales of cough and cold medications to analgesic sales (the latter being a proxy for total sales) using cyclical, linear regression models.
  • Another study (Magruder et al., 2004) examined the correlation between counts of OTC medications for influenza-like illness and community illness rates using Poisson regression analysis.
  • Using the cross-correlation function, one study (Hogan et al., 2003) examined the correlation between outbreaks detected from electrolyte solution sales data and outbreaks detected from hospital diagnoses (both using an EWMA to determine the start of the detected outbreaks).

Key Findings

  • Goldenberg and colleagues (2002) conclude that although OTC medication sales are “noisy” (i.e., they demonstrate a fair amount of fluctuation from day to day), they are a useful source of data for early detection of bioterrorist attacks. However, the researchers were not able to detect the onset of a local influenza epidemic.
  • The other studies concluded that OTC sales data are strongly correlated with clinical data (Das et al., 2005) and can indicate outbreaks of infectious diseases earlier than hospital discharge data can (Magruder et al., 2004), although the “earliness” varies by location of the outbreak and from year to year (Hogan et al., 2003).
  • Although outbreak detection is not directly relevant to our scope of BH impacts from PHEs, these studies illustrate the timeliness of OTC sales data and an ability to detect differences from baseline OTC sales.

Selected Publications Linking Emergency Department Visit Data with Behavioral Health or Public Health Emergencies

  • Joshua J. Baugh, Benjamin A. White, Dustin McEvoy, Brian J. Yun, David F. M. Brown, Ali S. Raja, and Sayon Dutta, “The Cases Not Seen: Patterns of Emergency Department Visits and Procedures in the Era of COVID-19,” American Journal of Emergency Medicine, Vol. 46, August 2021. https://www.ncbi.nlm.nih.gov/pubmed/33189517
  • Achintya N. Dey, Deborah Gould, Nelson Adekoya, Peter Hicks, Girum S. Ejigu, Roseanne English, Jenny Couse, and Hong Zhou, “Use of Diagnosis Code in Mental Health Syndrome Definition,” Online Journal of Public Health Informatics, Vol. 10, No. 1, 2018. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6088022/
  • Charles DiMaggio, Sandro Galea, and Lynne D. Richardson, “Emergency Department Visits for Behavioral and Mental Health Care After a Terrorist Attack,” Annals of Emergency Medicine, Vol. 50, No. 3, September 2007. https://www.ncbi.nlm.nih.gov/pubmed/17145111
  • Sidra Goldman-Mellor, Yusheng Jia, Kevin Kwan, and Jared Rutledge, “Syndromic Surveillance of Mental and Substance Use Disorders: A Validation Study Using Emergency Department Chief Complaints,” Psychiatric Services, Vol. 69, No. 1, January 1, 2018. https://pubmed.ncbi.nlm.nih.gov/28692388/
  • Kathleen P. Hartnett, Aaron Kite-Powell, Jourdan DeVies, Michael A. Coletta, Tegan K. Boehmer, Jennifer Adjemian, and Adi V. Gundlapalli, “Impact of the COVID-19 Pandemic on Emergency Department Visits—United States, January 1, 2019–May 30, 2020,” Morbidity and Mortality Weekly Report, Vol. 69, No. 23, June 12, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32525856
  • Fangtao Tony He, Nneka Lundy De La Cruz, Donald Olson, Sungwoo Lim, Amber Levanon Seligson, Gerod Hall, Jillian Jessup, and Charon Gwynn, “Temporal and Spatial Patterns in Utilization of Mental Health Services During and After Hurricane Sandy: Emergency Department and Inpatient Hospitalizations in New York City,” Disaster Medicine and Public Health Preparedness, Vol. 10, No. 3, June 2016. https://www.ncbi.nlm.nih.gov/pubmed/27292172
  • Kristin M. Holland, Christopher Jones, Alana M. Vivolo-Kantor, Nimi Idaikkadar, Marissa Zwald, Brooke Hoots, Ellen Yard, Ashley D’Inverno, Elizabeth Swedo, May S. Chen, et al., “Trends in US Emergency Department Visits for Mental Health, Overdose, and Violence Outcomes Before and During the COVID-19 Pandemic,” JAMA Psychiatry, Vol. 78, No. 4, April 1, 2021. https://www.ncbi.nlm.nih.gov/pubmed/33533876
  • Wei Hou, Elizabeth Brutsch, Angela C. Dunn, Cindy L. Burnett, Melissa P. Dimond, and Allyn K. Nakashima, “Using Syndromic Data for Opioid Overdose Surveillance in Utah,” Online Journal of Public Health Informatics, Vol. 10, No. 1, May 22, 2018. https://journals.uic.edu/ojs/index.php/ojphi/article/view/8988/7240
  • S. Janet Kuramoto-Crawford, Erica L. Spies, and John Davies-Cole, “Detecting Suicide-Related Emergency Department Visits Among Adults Using the District of Columbia Syndromic Surveillance System,” Public Health Reports, Vol. 132, No. 1, July–August 2017. https://pubmed.ncbi.nlm.nih.gov/28692388
  • Rebecca T. Leeb, Rebecca H. Bitsko, Lakshmi Radhakrishnan, Pedro Martinez, Rashid Njai, and Kristin M. Holland, “Mental Health–Related Emergency Department Visits Among Children Aged <18 Years During the COVID-19 Pandemic—United States, January 1–October 17, 2020,” Morbidity and Mortality Weekly Report, Vol. 69, No. 45, November 13, 2020. https://www.cdc.gov/mmwr/volumes/69/wr/mm6945a3.htm
  • Taylor A. Ochalek, Kirk L. Cumpston, Brandon K. Wills, Tamas S. Gal, and F. Gerard Moeller, “Nonfatal Opioid Overdoses at an Urban Emergency Department During the COVID-19 Pandemic,” JAMA, Vol. 324, No. 16, October 27, 2020. https://jamanetwork.com/journals/jama/fullarticle/2770986
  • Elizabeth Swedo, Nimi Idaikkadar, Ruth Leemis, Taylor Dias, Lakshmi Radhakrishnan, Zachary Stein, May Chen, Nickolas Agathis, and Kristin Holland, “Trends in U.S. Emergency Department Visits Related to Suspected or Confirmed Child Abuse and Neglect Among Children and Adolescents Aged <18 Years Before and During the COVID-19 Pandemic—United States, January 2019–September 2020,” Morbidity and Mortality Weekly Report, Vol. 69, No. 49, December 11, 2020. https://www.cdc.gov/mmwr/volumes/69/wr/mm6949a1.htm
  • Ellen Yard, Lakshmi Radhakrishnan, Michael F. Ballesteros, Michael Sheppard, Abigail Gates, Zachary Stein, Kathleen Hartnett, Aaron Kite-Powell, Loren Rodgers, Jennifer Adjemian, et al., “Emergency Department Visits for Suspected Suicide Attempts Among Persons Aged 12–25 Years Before and During the COVID-19 Pandemic—United States, January 2019–May 2021,” Morbidity and Mortality Weekly Report, Vol. 70, No. 24, June 18, 2021. https://pubmed.ncbi.nlm.nih.gov/34138833/

Study Objectives and Analytic Approaches

  • We identified studies that examined the use of ED records to conduct syndromic surveillance, to detect trends in ED visits for certain conditions before and during PHEs (in the case of COVID-19), or to compare ED visits for BH reasons before and after PHEs (in the case of several articles on Hurricane Sandy and one on the 2001 World Trade Center attack).
  • We also included articles that discuss methodological considerations that are relevant to our work, such as validating the use of ED chief complaints to conduct syndromic surveillance of mental health and substance use disorders (Goldman-Mellor et al., 2018), developing a mental health syndrome definition using diagnostic codes (Dey et al., 2018), and detecting suicide-related ED visits (Kuramoto-Crawford, Spies, and Davies-Cole, 2017) and those for opioid overdose using automatic classification methods (Hou et al., 2018).
  • Some of the articles reported using ED records through NSSP (Hartnett et al., 2020; Leeb et al., 2020). As with the data sources already described, these studies used widely variable analytic methods depending on their objectives.
  • Validation studies and studies of concordance between diagnoses assigned in the ED and those assigned at discharge from the hospital calculated measures of agreement, including percentage agreement and Cohen’s kappa statistic. Several studies included descriptive univariate and graphical approaches (e.g., trend plots) and bivariate analyses (chi-squared tests, z-tests), and most performed some type of regression modeling (e.g., a generalized linear Poisson model for count data, a generalized linear mixed effects model for longitudinal data, a joinpoint regression, or an ARIMA for autocorrelated data).

Key Findings

  • Taken together, the articles that examined the potential impact of various PHEs on ED visits demonstrated an ability to detect changes in several outcomes of interest.
  • One study demonstrated a 10.1-percent relative temporal increase in the rate of ED behavioral and mental health diagnosis after the 2001 World Trade Center attack (DiMaggio, Galea, and Richardson, 2007).
  • Another study demonstrated a 23-percent increase in psychiatric ED visits from patients living in the service areas of hospitals closed because of Hurricane Sandy (He et al., 2016).
  • Among several COVID-19–related studies we identified, one key finding was a decrease in total ED visit volume at the beginning of the COVID-19 pandemic (Baugh et al., 2021; Hartnett et al., 2020; Holland et al., 2021).
  • Another study described a 66-percent increase in mental health–related ED visits among children during two weeks in April 2020 compared with the same two weeks in April 2019 (Leeb et al., 2020).
  • Another finding was an increase in nonfatal opioid overdose visits, from 102 between March and June 2019 to 227 during the same period of 2020 (Ochalek et al., 2020).
  • Another finding was a 53-percent decrease in the number of ED visits related to child abuse and neglect during corresponding periods in 2020 compared with 2019 (Swedo et al., 2020).
  • For the studies validating the use of ED chief complaints for syndromic surveillance or examining concordance and agreement between ED and discharge diagnoses, authors found that, for example, using chief complaints alone may underestimate mental health–related ED visits, so discharge diagnosis data are useful to include in syndromic surveillance systems. Studies of diagnoses applied in the ED showed high concordance with discharge diagnoses after an inpatient hospitalization.

Selected Publications Linking Emergency Medical Services Activations with Behavioral Health or Public Health Emergencies

  • Miriam M. Calkins, Tania Busch Isaksen, Benjamin A. Stubbs, Michael G. Yost, and Richard A. Fenske, “Impacts of Extreme Heat on Emergency Medical Service Calls in King County, Washington, 2007–2012: Relative Risk and Time Series Analyses of Basic and Advanced Life Support,” Environmental Health, Vol. 15, No. 1, December 2016. https://ehjournal.biomedcentral.com/articles/10.1186/s12940-016-0109-0
  • Aubrey C. DeVine, Phuong T. Vu, Michael G. Yost, Edmund Y. W. Seto, and Tania M. Busch Isaksen, “A Geographical Analysis of Emergency Medical Service Calls and Extreme Heat in King County, WA, USA (2007–2012),” International Journal of Environmental Research and Public Health, Vol. 14, No. 8, August 2017. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5580639/
  • Mark Faul, Peter Lurie, Jeremiah M. Kinsman, Michael W. Dailey, Charmaine Crabaugh, and Scott M. Sasser, “Multiple Naloxone Administrations Among Emergency Medical Service Providers Is Increasing,” Prehospital Emergency Care, Vol. 21, No. 4, July–August 2017. https://pubmed.ncbi.nlm.nih.gov/28481656/
  • Nancy Glober, George Mohler, Philip Huynh, Tom Arkins, Dan O’Donnell, Jeremy Carter, and Brad Ray, “Impact of COVID-19 Pandemic on Drug Overdoses in Indianapolis,” Journal of Urban Health, Vol. 97, No. 6, December 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7529089/
  • Chris Kearns and Tanveer Islam, “Analysis of 9-1-1 Call Data from an Emergency Management Perspective: A Case Study of the City of Lethbridge,” Journal of Emergency Management, Vol. 16, No. 5, September 1, 2018. https://pubmed.ncbi.nlm.nih.gov/30387853
  • Utsha G. Khatri, Lia N. Pizzicato, Kendra Viner, Emily Bobyock, Monica Sun, Zachary F. Meisel, and Eugenia C. South, “Racial/Ethnic Disparities in Unintentional Fatal and Nonfatal Emergency Medical Services–Attended Opioid Overdoses During the COVID-19 Pandemic in Philadelphia,” JAMA Network Open, Vol. 4, No. 1, 2021. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2775360
  • Daniel L. Lemkin, Michael C. Bond, Donald W. Alves, and Richard A. Bissell, “A Public Health Enforcement Initiative to Combat Underage Drinking Using Emergency Medical Services Call Data,” Prehospital and Disaster Medicine, Vol. 27, No. 2, April 2012. https://pubmed.ncbi.nlm.nih.gov/22591633/
  • Heather A. Lindstrom, Brian M. Clemency, Ryan Snyder, Joseph D. Consiglio, Paul R. May, and Ronald M. Moscati, “Prehospital Naloxone Administration as a Public Health Surveillance Tool: A Retrospective Validation Study,” Prehospital and Disaster Medicine, Vol. 30, No. 4, August 2015. https://pubmed.ncbi.nlm.nih.gov/26061280/
  • Catherine Qualls, Hilary A. Hewes, N. Clay Mann, Mengtao Dai, and Kathleen Adelgais, “Documentation of Child Maltreatment by Emergency Medical Services in a National Database,” Prehospital Emergency Care, Vol. 25, No. 5, September–October 2021. https://pubmed.ncbi.nlm.nih.gov/32870747/
  • Elizabeth L. Seaman, Mathew J. Levy, J. Lee Jenkins, Cassandra Chiras Godar, and Kevin G. Seaman, “Assessing Pediatric and Young Adult Substance Use Through Analysis of Prehospital Data,” Prehospital and Disaster Medicine, Vol. 29, No. 5, 2014. https://www.cambridge.org/core/journals/prehospital-and-disaster-medicine/article/assessing-pediatric-and-young-adult-substance-use-through-analysis-of-prehospital-data/F87CAD08FAC6C895FBD2FA95BC51F9C1
  • Svetla Slavova, Peter Rock, Heather M. Bush, Dana Quesinberry, and Sharon L. Walsh, “Signal of Increased Opioid Overdose During COVID-19 from Emergency Medical Services Data,” Drug and Alcohol Dependence, Vol. 214, September 1, 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7351024/
  • Silas W. Smith, James Braun, Ian Portelli, Sidrah Malik, Glenn Asaeda, Elizabeth Lancet, Binhuan Wang, Ming Hu, David C. Lee, et al., “Prehospital Indicators for Disaster Preparedness and Response: New York City Emergency Medical Services in Hurricane Sandy,” Disaster Medicine and Public Health Preparedness, Vol. 10, No. 3, June 2016. https://www.ncbi.nlm.nih.gov/pubmed/26740248

Study Objectives and Analytic Approaches

  • The articles that were identified as using EMS data in novel ways as a PH surveillance tool typically examined spatial and temporal trends in EMS calls (i.e., 9-1-1 calls).
  • The most-common analytic methods found in these studies were descriptive statistics (frequencies and proportions across periods), spatial analyses, time series analyses (including segmented regression and ARIMA models), cross-correlational analyses, bivariate analyses, and logistic regression (to predict the characteristics of patients receiving multiple naloxone administrations by EMS providers) (Faul et al., 2017). Although this latter research topic is only indirectly related to the scope of our work, we included this study because it provided useful information about the variables contained in this data source.
  • For example, some studies (Calkins et al., 2016; DeVine et al., 2017) used time series analyses to examine associations between extreme heat events and EMS call volume.
  • Another study (Kearns and Islam, 2018) looked at spatial and temporal trends in 9-1-1 call data before, during, and after a wildfire. A different study (Lindstrom et al., 2015) used cross-correlation methods to examine the correlation between naloxone administration by EMS providers in the prehospital setting and the frequency of overdose-related ED visits. Still others used EMS data to “signal” something—such as substance use in youth in a single county in Maryland (Seaman et al., 2014), child maltreatment by EMS providers (Qualls et al., 2021), health system–level stress due to a New York City hospital closure during Hurricane Sandy (Smith et al., 2016), and the location of bars in a city that may be disproportionately serving alcohol to minors (by matching EMS call data to hospital records) (Lemkin et al., 2012). The most relevant study for our work, by Glober et al. (2020), used ARIMA for forecasting to examine changes in expected and observed daily rates of EMS calls (all calls for service, overdose calls, and calls in which naloxone was administered) before and after the Indianapolis stay-at-home order associated with COVID-19.

Key Findings

  • The key findings vary widely given the differing study objectives.
  • Relevant to this toolkit, the study validating the use of prehospital naloxone as a PH surveillance tool for opioid-related problems in a community found that the frequency of naloxone administration by EMS providers correlated in both the short and long terms with overdose-related ED visits (Lindstrom et al., 2015). However, the authors found frequent inconsistencies in the demographic data between these two sources.
  • Additionally, the study examining changes in daily number of Kentucky EMS runs for opioid overdose from January to April 2020, using data from the Kentucky State Ambulance Reporting System, found a 17-percent increase in EMS transports, a 71-percent increase in transport refusals, and a 50-percent increase in EMS runs for suspected opioid overdose with death at the scene (Slavova et al., 2020).
  • Similarly, Glober and colleagues (2020) found a 4-percent increase in all EMS calls for service, a 43-percent increase in calls for drug overdoses, and a 61-percent increase in calls in which naloxone was administered after the stay-at-home order due to COVID-19. In this study, there was no change in race or ethnicity of individuals with overdoses; in contrast, another study from Philadelphia (Khatri et al., 2021) found that COVID-19 was associated with increases in opioid overdose among non-Hispanic Black individuals compared with decreases among non-Hispanic White individuals.

Selected Publications Linking Social Media Data with Behavioral Health or Public Health Emergencies

  • Alaa Abd-Alrazaq, Dari Alhuwail, Mowafa Househ, Mounir Hamdi, and Zubair Shah, “Top Concerns of Tweeters During the COVID-19 Pandemic: Infoveillance Study,” Journal of Medical Internet Research, Vol. 22, No. 4, April 21, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32287039)170115
  • Corey H. Basch, Lorie Donelle, Joseph Fera, and Christie Jaime, “Deconstructing TikTok Videos on Mental Health: Cross-Sectional, Descriptive Content Analysis,” JMIR Formative Research, Vol. 6, No. 5, May 19, 2022. https://www.ncbi.nlm.nih.gov/pubmed/35588057
  • Daniel A. Bowen, Jing Wang, Kristin Holland, Brad Bartholow, and Steven A. Sumner, “Conversational Topics of Social Media Messages Associated with State-Level Mental Distress Rates,” Journal of Mental Health, Vol. 29, No. 2, April 2020. https://www.tandfonline.com/doi/full/10.1080/09638237.2020.1739251
  • Scott R. Braithwaite, Christophe Giraud-Carrier, Josh West, Michael D. Barnes, and Carl Lee Hanson, “Validating Machine Learning Algorithms for Twitter Data Against Established Measures of Suicidality,” JMIR Mental Health, Vol. 3, No. 2, May 16, 2016. https://www.ncbi.nlm.nih.gov/pubmed/27185366
  • Heather M. Brandt, Gabrielle Turner-McGrievy, Daniela B. Friedman, Danielle Gentile, Courtney Schrock, Tracey Thomas, and Delia West, “Examining the Role of Twitter in Response and Recovery During and After Historic Flooding in South Carolina,” Journal of Public Health Management & Practice, Vol. 25, No. 5, September–October 2019. https://www.ncbi.nlm.nih.gov/pubmed/31348171
  • David A. Broniatowski, Michael J. Paul, and Mark Dredze, “National and Local Influenza Surveillance Through Twitter: An Analysis of the 2012–2013 Influenza Epidemic,” PLoS One, Vol. 8, No. 12, 2013. https://www.ncbi.nlm.nih.gov/pubmed/24349542
  • Cynthia C. Chew and Gunther Eysenbach, “Pandemics in the Age of Twitter: Content Analysis of Tweets During the 2009 H1N1 Outbreak,” PLoS One, Vol. 5, No. 11, November 29, 2010. https://www.ncbi.nlm.nih.gov/pubmed/21124761
  • Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine, “Natural Language Processing of Social Media as Screening for Suicide Risk,” Biomed Informatics Insights, Vol. 10, 2018. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6111391
  • Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar, “Discovering Shifts to Suicidal Ideation from Mental Health Content in Social Media,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vol. 2016, May 2016. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5659860/pdf/nihms909062.pdf
  • Bruce Doré, Leonard Ort, Ofir Braverman, and Kevin N. Ochsner, “Sadness Shifts to Anxiety over Time and Distance from the National Tragedy in Newtown, Connecticut,” Psychological Science, Vol. 26, No. 4, April 2015. https://www.ncbi.nlm.nih.gov/pubmed/25767209
  • Johannes C. Eichstaedt, Robert J. Smith, Raina M. Merchant, Lyle H. Ungar, Patrick Crutchley, Daniel Preoţiuc-Pietro, David A. Asch, and H. Andrew Schwartz, “Facebook Language Predicts Depression in Medical Records,” Proceedings of the National Academy of Sciences of the United States of America, Vol. 115, No. 44, October 30, 2018. https://www.ncbi.nlm.nih.gov/pubmed/30322910
  • Candelaria Garcia, Jeovanna Amador Ayala, Kate Diaz Roldan, and Niloofar Bavarian, “Exploring Reddit Conversations About Mental Health Difficulties Among College Students During the COVID-19 Pandemic,” Journal of American College Health, August 24, 2022. https://www.ncbi.nlm.nih.gov/pubmed/36001484
  • Steven Gittelman, Victor Lange, Carol A. Gotway Crawford, Catherine A. Okoro, Eugene Lieb, Satvinder S. Dhingra, and Elaine Trimarchi, “A New Source of Data for Public Health Surveillance: Facebook Likes,” Journal of Medical Internet Research, Vol. 17, No. 4, April 20, 2015. https://www.jmir.org/2015/4/e98/
  • Elizabeth M. Glowacki, Gary B. Wilcox, and Joseph B. Glowacki, “Identifying #Addiction Concerns on Twitter During the COVID-19 Pandemic: A Text Mining Analysis,” Substance Abuse, Vol. 42, No. 1, 2021. https://www.ncbi.nlm.nih.gov/pubmed/32970973
  • Oliver Gruebner, Sarah R. Lowe, Martin Sykora, Ketan Shankardass, S. V. Subramanian, and Sandro Galea, “A Novel Surveillance Approach for Disaster Mental Health,” PLoS One, Vol. 12, No. 7, 2017. https://www.ncbi.nlm.nih.gov/pubmed/28723959
  • Oliver Gruebner, Martin Sykora, Sarah R. Lowe, Ketan Shankardass, Ludovic Trinquart, Tom Jackson, S. V. Subramanian, and Sandro Galea, “Mental Health Surveillance After the Terrorist Attacks in Paris," The Lancet, Vol. 387, No. 10034, May 28, 2016. https://www.ncbi.nlm.nih.gov/pubmed/27302026
  • Sharath Chandra Guntuku, Rachelle Schneider, Arthur Pelullo, Jami Young, Vivien Wong, Lyle Ungar, Daniel Polsky, Kevin G. Volpp, and Raina Merchant, “Studying Expressions of Loneliness in Individuals Using Twitter: An Observational Study,” BMJ Open, Vol. 9, No. 11, November 4, 2019. https://www.ncbi.nlm.nih.gov/pubmed/31685502
  • Sharath Chandra Guntuku, Garrick Sherman, Daniel C. Stokes, Anish K. Agarwal, Emily Seltzer, Raina M. Merchant, and Lyle H. Ungar, “Tracking Mental Health and Symptom Mentions on Twitter During COVID-19,” Journal of General Internal Medicine, Vol. 35, No. 9, September 2020. https://www.ncbi.nlm.nih.gov/pubmed/32638321
  • Shannon S. C. Herrick, Laura Hallward, and Lindsay R. Duncan, “‘This Is Just How I Cope’: An Inductive Thematic Analysis of Eating Disorder Recovery Content Created and Shared on TikTok Using #Edrecovery,” International Journal of Eating Disorders, Vol. 54, No. 4, April 2021. https://www.ncbi.nlm.nih.gov/pubmed/33382136
  • Timothy Y. Liu, Jason L. Sanders, Fu-Chiang Tsui, Jeremy U. Espino, Virginia M. Dato, and Joe Suyama, “Association of Over-the-Counter Pharmaceutical Sales with Influenza-Like-Illnesses to Patient Volume in an Urgent Care Setting,” PLoS One, Vol. 8, No. 3, 2013.
  • Daniel M. Low, Laurie Rumker, Tanya Talkar, John Torous, Guillermo Cecchi, and Satrajit S. Ghosh, “Natural Language Processing Reveals Vulnerable Mental Health Support Groups and Heightened Health Anxiety on Reddit During COVID-19: Observational Study,” Journal of Medical Internet Research, Vol. 22, No. 10, October 12, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32936777
  • May Oo Lwin, Jiahui Lu, Anita Sheldenkar, Peter Johannes Schulz, Wonsun Shin, Raj Gupta, and Yinping Yang, “Global Sentiments Surrounding the COVID-19 Pandemic on Twitter: Analysis of Twitter Trends,” JMIR Public Health and Surveillance, Vol. 6, No. 2, May 22, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32412418
  • Danielle Mowery, Hilary Smith, Tyler Cheney, Greg Stoddard, Glen Coppersmith, Craig Bryan, and Mike Conway, “Understanding Depressive Symptoms and Psychosocial Stressors on Twitter: A Corpus-Based Study,” Journal of Medical Internet Research, Vol. 19, No. 2, February 28, 2017. https://www.ncbi.nlm.nih.gov/pubmed/28246066
  • Prabhsimran Singh, Sukhjeet Singh, Manreet Sohal, Yogesh K. Dwivedi, Karanjeet Singh Kahlon, and Ravinder Singh Sawhney, “Psychological Fear and Anxiety Caused by COVID-19: Insights from Twitter Analytics,” Asian Journal of Psychiatry, Vol. 54, December 2020. https://www.ncbi.nlm.nih.gov/pubmed/32688277
  • Daniel C. Stokes, Anietie Andy, Sharath Chandra Guntuku, Lyle H. Ungar, and Raina M. Merchant, “Public Priorities and Concerns Regarding COVID-19 in an Online Discussion Forum: Longitudinal Topic Modeling,” Journal of General Internal Medicine, Vol. 35, No. 7, July 2020. https://www.ncbi.nlm.nih.gov/pubmed/32399912
  • Hong-Hee Won, Woojae Myung, Gil-Young Song, Won-Hee Lee, Jong-Won Kim, Bernard J. Carroll, and Doh Kwan Kim, “Predicting National Suicide Numbers with Social Media Data,” PLoS One, Vol. 8, No. 4, 2013. https://www.ncbi.nlm.nih.gov/pubmed/23630615

Study Objectives and Analytic Approaches

  • We found 25 relevant articles that used social media data, including Twitter and Facebook and emerging platforms, such as TikTok.
  • For Twitter data, the content of each tweet (e.g., frequency of words or phrases, such as a hashtag) was categorized and analyzed (Brandt et al., 2019; Chew and Eysenbach, 2010; Glowacki, Wilcox, and Glowacki, 2021). The studies we reviewed typically used daily or weekly search volume (often called relative search volume, normalized to the total number of searches in that same period) to estimate the prevalence of discussion around a specific topic. They also reported the time stamp, number of likes and retweets, and user profile information (e.g., number of followers and a Twitter-developed influence score for each user based on how frequently they were retweeted and followed by other users).
  • In a study aimed to establish the prevalence of discussion about a mental health topic or frequency of symptom presentation, the analytic sample was created by taking a random sample of U.S. tweets (e.g., a 1-percent random sample of daily U.S. tweets to look for mentions of symptoms of depression) (Guntuku et al., 2019; Guntuku et al., 2020; Mowery et al., 2017).
  • In looking for tweets referencing a specific PHE and BH issue (e.g., COVID-19 and addiction, Newton and depression), researchers conducted a targeted search for relevant tweets (Abd-Alrazaq et al., 2020; Doré et al., 2015).
  • One other method of creating an analytic sample involved sampling geotagged tweets within a certain radius around a PHE (e.g., a 15-kilometer radius around Paris after the 2015 terrorist attacks) (Gruebner et al., 2017; Gruebner et al., 2016).
  • A major limitation of this data source is that only a small number of Twitter users have or provide location data—and, even if the user associates their profile with a specific location, their tweet could be sent from a different location (e.g., while on vacation), resulting in misassignment to a location.
  • For analyses using Twitter and TikTok data, studies relied primarily on exploratory, descriptive analyses and sentiment analysis (Lwin et al., 2020; Singh et al., 2020; Basch et al., 2022; Herrick, Hallward, and Duncan, 2021; Garcia et al., 2022; Low et al., 2020). Given that social media data extracts are enormous, some studies examined automated classification (manual and automated coding assessing agreement) or applied machine learning methods (e.g., estimating suicide risk from social media data using deep learning methods) (Chew and Eysenbach, 2010; Coppersmith et al., 2018). Descriptive methods looked for patterns in the frequency of tweets about a particular topic. For example, two proportion z-tests were used in one study to compare changes in the prevalence of tweets about “fear and stress” associated with COVID-19), while other studies examined correlations between these patterns and other population-level data (e.g., rates of emotional distress on a self-reported survey) (Bowen et al., 2020; Braithwaite et al., 2016; Broniatowski, Paul, and Dredze, 2013; Won et al., 2013).
  • Facebook text data were analyzed to screen for language that suggested that the user was depressed (e.g., references to typical symptoms, including sadness, loneliness, hostility, rumination, and increased self-reference).
  • Studies also looked for patterns in post length, frequency of posting, time of day of posting, and number of “likes” received that could accurately predict a diagnosis of depression, as recorded in an electronic medical record (Gittelman et al., 2015).
  • Facebook has an advertising application programming interface that can calculate the number of “likes” that a specific post receives by zip code, which would support more-granular analyses (Eichstaedt et al., 2018).
  • Reddit threads (De Choudhury et al., 2016; Stokes et al., 2020), blogs, and other public discussion boards (also known as “subreddits”) related to mental health, suicide, and COVID-19 were also analyzed to determine, on a daily basis, which topics were being discussed most frequently (Coppersmith et al., 2018). Public discussion boards have variable information on users’ locations.

Key Findings

  • Most of the studies using Twitter data were conducted to describe a relationship between PHEs and their discussion on Twitter. For example, tweets geolocated in New York up to 11 days after Hurricane Sandy clustered around fear (Gruebner et al., 2017); similarly, geolocated tweets in Paris around the sites of the 2015 terrorist attack showed clusters of fear and sadness within the first two days after the attacks (Gruebner et al., 2016). However, we did not identify any studies that demonstrated the use of Twitter data to accurately predict mental health conditions or suicide risk.
  • One study using Facebook data had success in predicting depression with a similar level of accuracy of screening surveys benchmarked against medical records by analyzing posts for language about emotional (sadness), interpersonal (loneliness, hostility), and cognitive (preoccupation with the self, rumination) processes. Using Facebook data from six months prior to a depression diagnosis (as recorded in an electronic health record or its matched time for controls) yielded a prediction accuracy (i.e., area under the curve) of 0.72 (Eichstaedt et al., 2018).
  • Analyses of Reddit data on a COVID-19 discussion board showed sensitivity to changes in discussion patterns based on evolving impacts of the pandemic. Discussions about the “impact on daily life” topics showed “travel” peaking early and dropping throughout the month in March 2020 (3.2 percent for March 3–6 compared with 1.0 percent for March 28–31, p < 0.001) and concern regarding “personal finances” increasing (1.5 percent for March 3–6 compared with 2.1 percent for March 28–31, p = 0.003) (Stokes et al., 2020).

Selected Publications Linking Web Searches with Behavioral Health or Public Health Emergencies

  • John W. Ayers, Benjamin M. Althouse, Jon-Patrick Allem, Niels Rosenquist, and Daniel E. Ford, “Seasonality in Seeking Mental Health Information on Google,” American Journal of Preventive Medicine, Vol. 44, No. 5, May 2013. https://www.ncbi.nlm.nih.gov/pubmed/23597817
  • John W. Ayers, Eric C. Leas, Derek C. Johnson, Adam Poliak, Benjamin M. Althouse, Mark Dredze, and Alicia L. Nobles, “Internet Searches for Acute Anxiety During the Early Stages of the COVID-19 Pandemic,” JAMA Internal Medicine, Vol. 180, No. 12, December 1, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32832984
  • Anasse Bari, Aashish Khubchandani, Junzhang Wang, Matthias Heymann, and Megan Cofee, “COVID-19 Early-Alert Signals Using Human Behavior Alternative Data,” Social Network Analysis and Mining, Vol. 11, No. 1, 2021. https://www.ncbi.nlm.nih.gov/pubmed/33558823
  • Herman Anthony Carneiro and Eleftherios Mylonakis, “Google Trends: A Web-Based Tool for Real-Time Surveillance of Disease Outbreaks,” Clinical Infectious Diseases, Vol. 49, No. 10, November 15, 2009. https://www.ncbi.nlm.nih.gov/pubmed/19845471
  • Michael T. Ford, Andrew T. Jebb, Louis Tay, and Ed Diener, “Internet Searches for Affect-Related Terms: An Indicator of Subjective Well-Being and Predictor of Health Outcomes Across US States and Metro Areas,” Applied Psychology: Health and Well-Being, Vol. 10, No. 1, March 2018.
  • Emily A. Halford, Alison M. Lake, and Madelyn S. Gould, “Google Searches for Suicide and Suicide Risk Factors in the Early Stages of the COVID-19 Pandemic,” PLoS One, Vol. 15, No. 7, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32706835
  • Takeshi Hamamura and Christian S. Chan, “Anxious? Just Google It: Social Ecological Factors of Internet Search Records on Anxiety,” Emotion, Vol. 20, No. 8, December 2020. https://www.ncbi.nlm.nih.gov/pubmed/31414837
  • Michael Hoerger, Sarah Alonzi, Laura M. Perry, Hallie M. Voss, Sanjana Easwar, and James I. Gerhart, “Impact of the COVID-19 Pandemic on Mental Health: Real-Time Surveillance Using Google Trends,” Psychological Trauma, Vol. 12, No. 6, September 2020. https://www.ncbi.nlm.nih.gov/pubmed/32790441
  • Christian Karmen, Robert C. Hsiung, and Thomas Wetter, “Screening Internet Forum Participants for Depression Symptoms by Assembling and Enhancing Multiple NLP Methods,” Computer Methods and Programs in Biomedicine, Vol. 120, No. 1, June 2015. https://www.ncbi.nlm.nih.gov/pubmed/25891366
  • Duleeka Knipe, Hannah Evans, Amanda Marchant, David Gunnell, and Ann John, “Mapping Population Mental Health Concerns Related to COVID-19 and the Consequences of Physical Distancing: A Google Trends Analysis,” Wellcome Open Research, Vol. 5, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32671230
  • Duleeka Knipe, Hannah Evans, Mark Sinyor, Thomas Niederkrotenthaler, David Gunnell, and Ann John, “Tracking Online Searches for Emotional Wellbeing Concerns and Coping Strategies in the UK During the COVID-19 Pandemic: A Google Trends Analysis,” Wellcome Open Research, Vol. 5, No. 220, 2020. https://wellcomeopenresearch.org/articles/5-220
  • Duleeka Knipe, David Gunnell, Hannah Evans, Ann John, and Daisy Fancourt, “Is Google Trends a Useful Tool for Tracking Mental and Social Distress During a Public Health Emergency? A Time-Series Analysis,” Journal of Affective Disorders, Vol. 294, November 1, 2021. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8411666/
  • Yu-Hsuan Lin, Ting-Wei Chiang, and Yu-Lun Lin, “Increased Internet Searches for Insomnia as an Indicator of Global Mental Health During the COVID-19 Pandemic: Multinational Longitudinal Study,” Journal of Medical Internet Research, Vol. 22, No. 9, September 21, 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7508633/
  • Rebecca Ling and Joon Lee, “Disease Monitoring and Health Campaign Evaluation Using Google Search Activities for HIV and AIDS, Stroke, Colorectal Cancer, and Marijuana Use in Canada: A Retrospective Observational Study,” JMIR Public Health Surveillance, Vol. 2, No. 2, October 12, 2016. https://www.ncbi.nlm.nih.gov/pubmed/27733330
  • Christine Ma-Kellams, Flora Or, Ji Hyun Baek, and Ichiro Kawachi, “Rethinking Suicide Surveillance: Google Search Data and Self-Reported Suicidality Differentially Estimate Completed Suicide Risk,” Clinical Psychological Science, Vol. 4, 2016. https://journals.sagepub.com/doi/pdf/10.1177/2167702615593475
  • Nir Menachemi, Saurabh Rahurkar, and Mandar Rahurkar, “Using Web-Based Search Data to Study the Public’s Reactions to Societal Events: The Case of the Sandy Hook Shooting,” JMIR Public Health Surveillance, Vol. 3, No. 1, March 23, 2017. https://www.ncbi.nlm.nih.gov/pubmed/28336508
  • Błażej Misiak, Dorota Szcześniak, Leszek Koczanowicz, and Joanna Rymaszewska, “The COVID-19 Outbreak and Google Searches: Is It Really the Time to Worry About Global Mental Health?” Brain, Behavior, and Immunity, Vol. 87, July 2020. https://www.ncbi.nlm.nih.gov/pubmed/32360605
  • Jason Parker, Courtney Cuthbertson, Scott Loveridge, Mark Skidmore, and Will Dyar, “Forecasting State-Level Premature Deaths from Alcohol, Drugs, and Suicides Using Google Trends Data,” Journal of Affective Disorders, Vol. 213, April 15, 2017. https://www.ncbi.nlm.nih.gov/pubmed/28171770
  • Ariadna Ramos-Gomez, Aldo A. Pérez-Escatel, Elio Atenógenes Villaseñor-García, and Cesar Ramos-Remus, “An Infodemiology Approach to Assess the Impact of Unemployment on Anxiety and Depression in France,” Forum for Social Economics, Routledge, 2021. https://www.tandfonline.com/doi/full/10.1080/07360932.2021.1880461
  • Jacob E. Simmering, Linnea A. Polgreen, and Philip M. Polgreen, “Web Search Query Volume as a Measure of Pharmaceutical Utilization and Changes in Prescribing Patterns,” Research in Social and Administrative Pharmacy, Vol. 10, No. 6, November–December 2014. https://www.ncbi.nlm.nih.gov/pubmed/24603135
  • Mark Sinyor, Matthew J. Spittal, and Thomas Niederkrotenthaler, “Changes in Suicide and Resilience-Related Google Searches During the Early Stages of the COVID-19 Pandemic,” Canadian Journal of Psychiatry, Vol. 65, No. 10, October 2020. https://www.ncbi.nlm.nih.gov/pubmed/32524848
  • Katerina Standish, “COVID-19, Suicide, and Femicide: Rapid Research Using Google Search Phrases,” Journal of General Psychology, Vol. 148, No. 3, July 2021. https://www.ncbi.nlm.nih.gov/pubmed/33480333
  • T. Vargas, J. Schiffman, P. H. Lam, A. Kim, and V. A. Mittal, “Using Search Engine Data to Gauge Public Interest in Mental Health, Politics and Violence in the Context of Mass Shootings,” PLoS One, Vol. 15, No. 8, 2020. https://www.ncbi.nlm.nih.gov/pubmed/32764767
  • Elad Yom-Tov, Ryen W. White, and Eric Horvitz, “Seeking Insights About Cycling Mood Disorders via Anonymized Search Logs,” Journal of Medical Internet Research, Vol. 16, No. 2, February 25, 2014. https://www.ncbi.nlm.nih.gov/pubmed/24568936

Study Objectives and Analytic Approaches

  • We identified more than 20 articles that examined trends in the number of web searches—or the relative number of searches (Carneiro and Mylonakis, 2009; Knipe, Evans, Marchant, et al., 2020; Knipe et al., 2021; Ling and Lee, 2016; Misiak et al., 2020)—using mental health–related terms (e.g., afraid, anxiety, depression, fatigue, fear, lonely, nervous, scared, sleepy, stress, tired, energetic, enthusiastic, happy, and strong), suicide-related terms (e.g., how to kill yourself, painless suicide), and mental health help-seeking terms (e.g., counselor).
  • In a study using Google Trends data, Comscore estimates were used to calculate raw counts of searches (Ayers et al., 2020). Some studies, including the illustrative study cited here, used natural language processing methods to break the textual information into its grammatical units (Karmen, Hsiung, and Wetter, 2015). Most analyses were performed in the context of COVID-19, focused on a small period (e.g., a single month) immediately after an event (e.g., first death from COVID-19), and did not report the data geographically at a sub-state level.
  • These studies used two main analytic approaches. Some studies compared web search data with data collected from a population-based survey to determine whether changes in web searching were correlated with other trends (e.g., self-reported symptoms of anxiety, deaths by suicide, COVID-19 cases) (Hamamura and Chan, 2020; Knipe, Evans, Sinyor, et al., 2020; Ma-Kellams et al., 2016; Parker et al., 2017; Yom-Tov, White, and Horvitz, 2014). Other studies looked at changes in the number or relative number of web searches over time using regression (Ayers et al., 2013; Menachemi, Rahurkar, and Rahurkar, 2017; Ramos-Gomez et al., 2021; Sinyor, Spittal, and Niederkrotenthaler, 2020; Standish, 2021; Vargas et al., 2020), wavelet phase analysis (Ayers et al., 2013), or ARIMA modeling (to predict the expected number of searches from past search behavior compared with the actual number of searches over time) (Halford, Lake, and Gould, 2020; Hoerger et al., 2020; Liu et al., 2013).
  • For example, one study used Google Trends data—which are available at the sub-state level, free of charge, and up-to-date as of the past hour—to explore trends in well-being at the state and metro levels (Ford et al., 2018). Web searches were analyzed for terms indicating negative or positive affect to determine the proportion of searches for each. Finally, one study explored isolation during COVID-19 by creating a net “movement index” comprising two types of web searches: (1) using terms referring to activities at home and (2) using terms involving mobility (e.g., gas stations, flight tickets). The authors examined the correlations between a lag analysis of the net movement index and the weekly change in COVID-19 cases (Bari et al., 2021).

Key Findings

  • Overall, studies found that changes in the volume of web searches were associated with changes in other population-level data.
    • Drug use: Search volume for specific drug names (e.g., amoxicillin, olopatadine) correlated closely with prescription drug use rates for infections and allergies (Simmering, Polgreen, and Polgreen, 2014); although not specific to BH, these same methods could be adopted to explore the use of prescriptions for medications related to BH.
    • Suicide: Google Trends data explain the substantial variation in growth of state-level rates of death (e.g., 21.8 percent for suicide rates) (Parker et al., 2017).
    • Anxiety: Google search rates for anxiety were associated with self-reported anxiety, and anxiety search rates increased following a major disaster (Hamamura and Chan, 2020).
    • Depression: Negative affect in web searches was correlated with symptoms of depression (in the most recent BRFSS) and indicators of well-being (in the most recent Gallup survey) (Ford et al., 2018).
    • Isolation: One study found that the net movement index correlated with weekly COVID-19 new case growth rate with a lag of between ten and 14 days for the United States at large, as well as at the state level for 42 out of 50 states from March to June 2020 (Bari et al., 2021).
    • General mental health: The relative number of searches for suicide and depression was negatively correlated with the number of COVID-19 cases and deaths (Misiak et al., 2020).
  • Studies also showed that the volume of web searches related to mental health and mental health help-seeking increased in response to a PHE and followed expected seasonal variation. For instance, searches for the Disaster Distress Helpline were remarkably elevated in the context of COVID-19 (Halford, Lake, and Gould, 2020).
  • Acute anxiety queries were cumulatively 11 percent (95-percent CI, 7 percent–14 percent) higher than expected for the 58-day period after COVID-19 was first declared a national emergency (March 13, 2020) (Ayers et al., 2020). Finally, mental health searches followed seasonal patterns, with winter peaks and summer troughs amounting to a 14-percent (95-percent CI, 11-percent–16-percent) difference in search volume in the United States (Ayers et al., 2013).

Selected Publications Linking Crisis Text Line Data with Behavioral Health or Public Health Emergencies

  • Margaret M. Sugg, P. Grady Dixon, and Jennifer D. Runkle, “Crisis Support-Seeking Behavior and Temperature in the United States: Is There an Association in Young Adults and Adolescents?” Science of the Total Environment, Vol. 669, June 15, 2019. https://www.ncbi.nlm.nih.gov/pubmed/30884264
  • Margaret M. Sugg, Kurt D. Michael, Scott E. Stevens, Robert Filbin, Jaclyn Weiser, and Jennifer D. Runkle, “Crisis Text Patterns in Youth Following the Release of 13 Reasons Why Season 2 and Celebrity Suicides: A Case Study of Summer 2018,” Preventive Medicine Reports, Vol. 16, December 2019. https://www.sciencedirect.com/science/article/pii/S2211335519301706
  • Hannah Selene Szlyk, Kimberly Beth Roth, and Víctor García-Perdomo, “Engagement with Crisis Text Line Among Subgroups of Users Who Reported Suicidality,” Psychiatric Services, Vol. 71, No. 4, 2019. https://ps.psychiatryonline.org/doi/10.1176/appi.ps.201900149
  • Laura K. Thompson, Kurt D. Michael, Jennifer Runkle, and Margaret M. Sugg, “Crisis Text Line Use Following the Release of Netflix Series 13 Reasons Why Season 1: Time-Series Analysis of Help-Seeking Behavior in Youth,” Preventive Medicine Reports, Vol. 14, 2019. https://www.sciencedirect.com/science/article/pii/S2211335519300154
  • Laura K. Thompson, Margaret M. Sugg, and Jennifer R. Runkle, “Adolescents in Crisis: A Geographic Exploration of Help-Seeking Behavior Using Data from Crisis Text Line,” Social Science and Medicine, Vol. 215, 2018. https://www.sciencedirect.com/science/article/pii/S0277953618304490

Study Objectives and Analytic Approaches

  • Among five studies we identified that used crisis text line (CTL) data, each took a slightly different approach.
  • The two studies that examined whether the count of CTL conversations in a certain period for a certain geographic area changes after a particular event (i.e., the release of 13 Reasons Why, celebrity suicides—Sugg, Dixon, and Runkle, 2019; Thompson et al., 2019) used interrupted time series and autoregressive integrated moving average (ARIMA) modeling.
  • A study of the association between unseasonably high ambient temperatures and CTL conversations (Sugg et al., 2019) used distributed lag non-linear regression modeling to examine the short-term effects of daily maximum (and minimum) temperature on crisis help-seeking behavior.
  • The other studies, which have less direct relevance to BH surveillance but demonstrate the strengths and limitations of this data source, used latent class analysis and spatial error regression modeling with pairwise multiple comparison tests to identify classes of CTL users (Szlyk, Roth, and García-Perdomo, 2019) and predictors of CTL usage (Thompson, Sugg, and Runkle, 2018), respectively.

Key Findings

  • In a study that used the ARIMA method, the authors detected a statistically significant excess number of crisis conversations over the expected volume after the release of both seasons of the television show 13 Reasons Why (Sugg et al., 2019).
  • The study using the distributed lag non-linear modeling approach detected a significant increase in CTL volume in the settings of higher maximum temperatures during the warm season and higher minimum temperatures in the cool season (Sugg, Dixon, and Runkle, 2019).