E: Exploratory Analyses of and Real-World Examples for the Most-Promising Data Sources

Unemployment Insurance Claims

This exploratory analysis demonstrates how you could monitor and assess for changes in unemployment trends following PHEs using UI claims data. We examined weekly initial UI claims data for Los Angeles (L.A.) County, which is the most populous county in California. We obtained data from the California Economic Development Department.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• Counts of weekly initial UI claims in L.A. County ranged from 7,749 to 13,939 claims per week over the years 2018 and 2019 (median = 10,796, mean = 39,767) to a maximum of 283,467 claims for the week ending March 28, 2020 (Analysis 1).
• Ten small California counties had weekly claims counts of zero (i.e., Alpine, Colusa, Inyo, Lassen, Mariposa, Modoc, Mono, Plumas, Sierra, and Trinity). There was a nearly equal share of zero-count weeks in each of the pre–COVID-19 years in our data (218 zero-count weeks in 2018; 244 zero-count weeks in 2019).
• The California Economic Development Department suppresses data for county-weeks that have counts of less than ten and not equal to zero. For the total initial UI claims counts by county and week, there were no suppressed counts.
• Counts of weekly initial UI claims broken down by gender included suppressed counts due to small sample size (< 10) for some counties. However, for each county, the maximum upper bound for the percentage of cases suppressed over the 176 county-weeks was less than 0.1 percent (with the exception of two counties).¹ The preponderance of suppressed weekly counts occurred in the “unknown” gender strata, which is the group of respondents who did not report gender. We estimated that a negligible percentage (less than 0.5 percent) of the individuals reporting claims did not report gender.

Visualizing Trends in Unemployment, Overall and by Subgroup

Analysis 1 shows weekly trends in UI claims for L.A. County. These data allow us to understand the pre-COVID-19 trends and identify the direction and magnitude of changes in these trends after two separate and overlapping PHEs of differing magnitudes (which affect all of the population of California, as opposed to just some): the California governor’s emergency declaration for COVID-19 (March 4, 2020) and the California wildfires (October 16, 2020).

Initial UI claims in Los Angeles were essentially flat in 2020 prior to the statewide declaration of COVID-19 emergency (March 4, 2020). After the state mandated business closures on March 19, 2021, claims peaked in the following week (week 13) at a record level. Claims stabilized over the next few months, then peaked again just before the October 16, 2020, disaster declaration (week 36) for the wildfires affecting L.A. County from September 4-November 17, 2020.

Key Takeaways

Initial claims were essentially flat prior to the statewide declaration of emergency (week 10; March 4, 2020). After the state mandate of business closures (March 19, 2020), claims peaked in the following week (week 13) at a record level. Claims stabilized over the next few months, then peaked again just before the October 16, 2020, disaster declaration (week 36) for the wildfires affecting L.A. County from September 4 to November 17, 2020.

In response to these trends, PH agencies could share this information with partners that provide employment-related services so that those partners could prepare for a likely increase in claims to process. Because unemployed individuals may also lose their health insurance, PH agencies may also need to plan for an increase in Medicaid applications or assisting the public with insurance marketplace enrollment. Lastly, given the negative BH effects of unemployment, PH agencies could coordinate with partners that handle unemployment claims to offer promotional materials, screenings, or services related to depression and substance use.

We next examine the trends in UI claims by gender, starting with Analysis 2, which displays the trends in total claims for men and women.

The time trend in the distribution of initial UI claims in L.A. County in 2020 for men and women separately is similar to the trend observed overall. Women peaked at a higher level than men in week 13, just after the COVID-19 emergency declaration, but men peaked at a higher level than women during the wildfires in week 36.

Key Takeaways

The time trend in the distribution of claims for men and women is similar to the trend observed overall. Women peaked at a higher level than men in week 13, just after the COVID-19 emergency declaration, but men peaked at a higher level than women during the wildfires in week 36.

To better visualize differences in patterns of UI claims for men and women, we calculated the relative gender distribution of claims over time (e.g., percentage female and percentage male, which total 100 percent). Analysis 3 displays the percentage of UI claims made by women throughout 2020 and the first part of 2021.

Prior to the COVID-19 emergency declaration, more men made initial UI claims than women, i.e., slightly less than 60% of the claims were from men and slightly more than 40% were from women. Following the COVID-19 emergency declaration, the gender distribution of claims equalized. It deviated little from about a 50-50 split through the remainder of the 2020 and into 2021, with one exception. During the wildfires, the gender distribution tipped back towards nearly the same unequal proportion of more men than women making claims.

Key Takeaways

Prior to the COVID-19 emergency declaration, more men made initial UI claims than women (i.e., slightly less than 60 percent of the claims were from men and slightly more than 40 percent were from women). Following the COVID-19 emergency declaration, the gender distribution of claims equalized. It deviated little from about a 50-50 split through the remainder of the year and into the following, with one exception: During the wildfires, the gender distribution tipped back toward nearly the same unequal proportion of more men than women making claims. These findings are consistent with evidence from national data that COVID-19 has affected women’s labor market participation more than men’s (Karageorge, 2020).

Should a specific gender group experience disproportionate increases in unemployment rates compared with another gender, this could inform targeted outreach approaches to subpopulations that are disproportionately experiencing unemployment challenges.

Assessing for a Significant Change After a Public Health Emergency

To assess whether the trends we saw in the data represented a statistically significant change in initial UI claims from the pre-PHE baseline, we conducted two types of modeling: ITSA and DiD analysis.

Interrupted Time Series Analysis

Using total initial UI claims data for L.A. County from 2018 to 2021, we employed the ITSA to examine the change in the level and the slope (or rate of change) in initial UI claims from before the COVID-19 emergency declaration to after and to conduct a preliminary examination of how the California wildfires departed from this trend.

The two modeling decisions in fitting a simple (pre-post) ITSA involve the date for the interruption in the time series and how many lags (periods) are required to fit the correlation structure of the data. For COVID-19, the descriptive data indicated that the time series peaked shortly after the business closure mandate (March 19, 2020; see Analysis 4); findings from using the earlier emergency declaration date were qualitatively similar. Additionally, we determined that at least two lags were required to model the correlation structure.²

In Analysis 4, we depict the predicted values and 95-percent confidence interval (CI) from an ITSA model with two lags, testing for an interruption in the time series during the week containing the business closure mandate.

Prior to COVID-19, initial UI claims in L.A. County were, on average, about 11,000 claims per week . In the week of the closure mandate, the claims increased by nearly 142,000 claims per week (141,900; 95-percent CI: 107,200, 176,600). There was a statistically significant downward trend following this peak. Claims counts spiked again for four weeks concurrent with the wildfires. These PHEs were associated with deviations from predicted claims rates.

Key Takeaways

Prior to COVID-19, initial UI claims were, on average, about 11,000 claims per week (mean = 11,037; 95-percent CI: 10,490, 11,585), with no statistically significant weekly change (i.e., slope = −3; 95-percent CI: −13, 7). In the week of the closure mandate, the claims increased by nearly 142,000 claims per week (141,900; 95-percent CI: 107,200, 176,600). There was a statistically significant downward trend following this peak. The claims counts for four weeks concurrent with the wildfires exceeded the upper bound of the 95-percent CI around this downward trend.

PH agencies may find it helpful to understand how multiple overlapping PHEs are affecting unemployment levels in their jurisdictions to inform targeted outreach and public awareness efforts around preventing downstream BH impacts of this major stressor.

The simple ITSA model in Analysis 4 could be extended to incorporate the changing treatment periods from the peak after the business closure mandate through to the wildfires and the months following that second PHE. However, ITSA is best designed to study sequential separate treatment periods. It is less suited to characterize how a second PHE may compound an ongoing PHE. As detailed in the next section, DiD analysis may be more optimal in this context.

Difference-in-Differences Analysis

We performed a DiD analysis (Table 1) using the weekly initial UI claims counts for each California county from 2018 to 2021 to distinguish peak COVID-19 impacts from long-term COVID-19 impacts and to evaluate the extent to which the wildfire PHE compounded long-term impacts of the COVID-19 PHE. We fit a DiD analysis with fixed effects for place and time, included adjustment for seasonality, and weighted the data in relationship to the population size.³

To examine the impact of potentially compounding PHEs, we evaluated the difference in the number of claims between the pre-COVID-19 period (i.e., all weeks prior to the March 19, 2020, mandate) and the post-COVID-19 period (i.e., all weeks including and after March 19, 2020) and the potential change in this pre-post difference through different stages of the overlapping PHEs. Specifically, we evaluated whether the pre-post COVID-19 effect was larger during

• the COVID-19 peak period (i.e., identified descriptively in Analysis 4) than during the non-peak periods
• the California wildfires than during non-wildfire periods.⁴

Predicted Differences in Initial Unemployment Insurance Claims: California Counties, 2018--2021

Difference Descriptions	Predicted Claims	95-Percent CI
Difference
COVID-19 difference (post-pre), not peak, not during wildfires	5,426	2,005	8,847
COVID-19 difference (post-pre), peak cases	15,307	5,191	25,423
COVID-19 difference (post-pre), wildfire	8,170	−270	16,609
DiD
(Post-pre difference, COVID-peak)–(post-pre difference, non-peak)	9,881	3,156	16,606
(Post-pre difference, wildfire)–(post-pre difference, non-wildfire)	2,744	−2,739	8,226

Note: This predicted difference is made for the non-peak and non-wildfire period. When including the peak in the post-COVID-19 period, the overall average is slightly higher, with an increase of 5,796 claims per week for the post-COVID-19 period versus the pre-COVID-19 period (95-percent CI: 2,042, 9,551).

Key Takeaways

The impact of COVID-19 was an increase of 5,426 additional weekly claims (95-percent CI: 2,005, 8,847) greater than the pre-COVID-19 period weekly claims count (of 550 claims, not shown; 95-percent CI: −935, 2,035). The pre-post difference during the peak-COVID-19 period was 15,307 additional weekly claims (95-percent CI: 5,191, 25,423) over the pre-COVID-19 period; during the wildfires, the difference was an increase of 8,170 additional weekly claims (95-percent CI: −270, 16,609) over the pre-COVID-19 period. All estimates adjust for seasonality and county differences. The pre-post increase in claims during the COVID-19-peak period (9,881 additional weekly claims, 95-percent CI: 3,156, 16,606) was larger than the pre-post increase in claims associated with the wildfires (2,744 additional weekly claims, 95-percent CI: −2,739, 8,226). In the context of higher unemployment levels due to the ongoing pandemic, the peak in COVID-19 cases had a much greater (and more significant) impact on weekly UI claims than the wildfires did.

Examples of Using Unemployment Insurance Initial Claims Data for Ongoing Monitoring

The data analyses reported in the previous section (i.e., ITSA; DiD analysis) provide a statistical modeling framework for understanding what the history of UI claims has been and identifying statistical thresholds for differences or deviations from this pattern. One approach for setting an ongoing monitoring threshold is to examine the distribution or variability of the data and identify an upper and lower threshold for values of the outcome.

As an example, Analysis 5 displays the UI claims data from May 2020 through the end of the year. This demonstrates one option for determining an alert threshold in the context of two overlapping PHEs. In this case, the pre-COVID-19 UI claims levels could have been used to set the alert threshold. However, with such a long-lasting PHE, we demonstrate another option that may prove more useful for PH officials. We set the alert threshold set based on claims data from May through June 2020, which (as Analysis 5 shows) was a period of relative stability in UI claims from week to week. Claims had reached a new status quo, and, rather than using data from before the pandemic, we set an alert threshold at the mean of weekly claims for this new status quo plus two times the standard deviation of claims for the period.

Around week 19 of 2020 and throughout the rest of May, initial UI claims began to stabilize around a value of about 100,000. A PH agency conducting ongoing monitoring would use this new baseline to set an alert threshold going forward of just over 150,000 weekly claims.

Key Takeaways

Following the COVID-19 peak, a PH agency conducting ongoing monitoring would notice that, around week 19 and throughout the rest of May 2020, initial UI claims began to stabilize around a value of about 100,000. Using this new baseline to set an alert threshold going forward would yield an alert of just over 150,000 weekly claims.

It is clear from the figure that UI claims, which were already elevated in the context of the ongoing pandemic, again spiked around the wildfires (particularly around week 35, ending August 29, 2020). This spike occurred one week after the governor declared a state of emergency and the President issued a disaster declaration. This example demonstrates why it is important for PH agencies to consider the recent context of ongoing emergencies when setting alert thresholds. In this case, using an older threshold (i.e., pre-COVID-19) would obscure the additive effect of the wildfires on unemployment rate.

Access selected publications that have used this data source for monitoring and surveillance.

2-1-1 Calls

For this exploratory analysis (the findings of which also appear in Disaster Medicine and Public Health Preparedness —see Landis et al., 2023), we examined 2-1-1 calls made in Broward County, Florida, around Hurricane Irma (September 4–11, 2017) and the start of COVID-19 (March 2020) to demonstrate (1) that you can detect a “signal” in the context of PHEs and (2) how you could monitor, and assess for anomalies in, trends in 2-1-1 calls.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• For Broward County in our period of interest, we had a sample size of 275,856 total calls, with an average weekly call volume of 1,591 (range: 866 to 2,733).
• Age is missing for 54 percent of calls; when reported, average age of callers is 45 years. Gender information is missing for less than 1 percent of calls; 73 percent of calls are by women. Race/ethnicity information is missing for 68 percent of calls; 16 percent of calls were made by White individuals, 15 percent by Black individuals.
• Call reason is missing for 20 percent of calls; 14 percent of calls were BH-related, 39 percent of calls were for “basic needs,” and 23 percent of calls were for “government/ community services.” Note that many calls had multiple reasons, so these categories are not mutually exclusive.
• If your county uses the standard 2-1-1 taxonomy for documenting the caller’s needs and the reason is rarely missing, you could identify BH-related calls using the level 1 code “R,” which corresponds to “Mental Health and Substance Use Disorder Services.” If your county does not use the standard taxonomy of call needs, you could tabulate the call reasons and determine which calls should be flagged as potentially BH-related (e.g., calls with the following major needs: “mental health,” “substance use,” and “suicide prevention/intervention”).

Visualizing Trends in 2-1-1 Calls, Overall and by Subgroup

A simple time series plot of the seven-day moving average of total 2-1-1 call volume in Broward County allows us to examine overall trends, relationships to the key PHEs of interest, and any unusual values or potential outliers (Analysis 6). We examined whether the trends we were seeing in 2-1-1 call volume varied by gender (Analysis 7).

Hurricane Irma in September 2017 was temporally associated with an increase in the seven-day moving average of total 2-1-1 call volume in Broward County. Calls peaked 17 days after the start of the hurricane; at this point, the seven-day moving average of total 2-1-1 calls was approximately 108 calls above the moving average for the same date in 2016. Total call volume returned to pre-hurricane levels in early October 2017, suggesting that call volume was elevated for approximately four weeks following this PHE.

Key Takeaways

Hurricane Irma was temporally associated with an increase in the seven-day moving average of total 2-1-1 call volume in Broward County. Calls peaked 17 days after the start of the hurricane; at this point, the seven-day moving average of total 2-1-1 calls was approximately 108 calls above the moving average for the same date in 2016. Total call volume returned to pre-hurricane levels in early October 2017, suggesting that call volume was elevated for approximately four weeks following the disaster.

Knowledge of these prior increases in call volumes in the PHE context could help PH agencies work with 2-1-1 call centers to be prepared to handle increased call volume and meet increased demand for services, such as referrals to food banks and mental health support.

Hurricane Irma was associated with an increase in the seven-day moving average of total 2-1-1 call volume for both men and women in Broward County. Calls by women peaked 17 days after the start of the hurricane; calls by men peaked eight days after the start of the hurricane. At their peak, calls by women and men were approximately 115 calls and six calls above the moving average for the same date in 2016, respectively.

Key Takeaways

When analyzed by gender, Hurricane Irma was associated with an increase in the seven-day moving average of total 2-1-1 call volume for both men and women in Broward County. Calls by women peaked 17 days after the start of the hurricane; calls by men peaked eight days after the start of the hurricane. At their peak, calls by women and men were approximately 115 calls and six calls above the moving average for the same date in 2016, respectively. These differences suggest that although 2-1-1 calls for both women and men increased in the wake of the hurricane, the increase was steeper and delayed for women relative to men.

Given the greater increases in calls among women, PH agencies, 2-1-1 call centers, and their community partners could, in their pre-PHE planning, ensure that they are prepared to address issues that disproportionately affect women, such as domestic violence, anxiety and depression, and food insecurity.

Similarly, Analysis 8 shows overall trends in 2-1-1 calls surrounding the start of the COVID-19 pandemic, and Analysis 9 shows volume of calls surrounding this PHE stratified by gender.

The start of the COVID-19 pandemic was associated with a large increase in the seven-day moving average of total 2-1-1 call volume in Broward County. Calls peaked two weeks after the start of the pandemic; at this point, the seven-day moving average of total 2-1-1 calls was approximately 104 calls above the moving average for the same date in 2019. Although the seven-day moving average of total calls after the start of the pandemic returned to pre-pandemic levels in late May 2020, the marked uptick in calls in the weeks after the start of the pandemic caused the average number of calls in the post-pandemic period to be greater than that for the pre-pandemic period.

Key Takeaways

The start of the COVID-19 pandemic was associated with a large increase in the seven-day moving average of total 2-1-1 call volume in Broward County. Calls peaked two weeks after the start of the pandemic; at this point, the seven-day moving average of total 2-1-1 calls was approximately 104 calls above the moving average for the same date in 2019. Although the seven-day moving average of total calls after the start of the pandemic returned to pre-pandemic levels in late May 2020, the marked uptick in calls in the weeks after the start of the pandemic caused the average number of calls in the post-pandemic period to be greater than that for the pre-pandemic period.

The start of the COVID-19 pandemic was associated with a large increase in the seven-day moving average of total 2-1-1 call volume for both men and women in Broward County. Calls by women peaked two weeks after the start of the pandemic; calls for men peaked three weeks after the start of the pandemic. At their peak, calls by women and men were approximately 59 calls and 49 calls above the moving average for the same date in 2019, respectively.

Key Takeaways

When analyzed by gender, the start of the COVID-19 pandemic was associated with a large increase in the seven-day moving average of total 2-1-1 call volume for both men and women in Broward County. Calls by women peaked two weeks after the start of the pandemic; calls for men peaked three weeks after the start of the pandemic. At their peak, calls by women and men were approximately 59 calls and 49 calls above the moving average for the same date in 2019, respectively. These differences suggest that although 2-1-1 calls for both women and men increased after the start of the pandemic, the increase was both slightly steeper and earlier for women relative to men.

These findings may signal to PH agencies that BH challenges may emerge at different times for different parts of the population.

Assessing for a Significant Change After a Public Health Emergency

To assess whether the trends we noted when visualizing the data represented a statistically significant change in 2-1-1 calls from the pre-PHE baseline, we conducted two types of modeling: ARIMA and ITSA.

ARIMA Modeling

Because of 2-1-1 call data’s strong weekly autocorrelation (i.e., daily call volume is highly correlated with the call volume on the same day in the prior week), we used an ARIMA model to examine whether the observed 2-1-1 call volumes after Hurricane Irma differed significantly from predicted values based on pre-hurricane trends. We decided on a simple ARIMA model without differencing (d = 0) that predicts current daily call volume using call volume over the past seven days and the forecasted error from the previous day. An important strength of ARIMA models is that they are dynamic: They constantly update predictions using user-specified history of both input and output values.

Analysis 10 shows the observed versus predicted values of daily call volume (top panel) and the standardized model residuals (i.e., the standardized difference between the observed and predicted values). Dates on which the observed number of calls exceeded the upper bound of the 99-percent CI of the model-predicted value of daily call volume are denoted by purple diamonds; on these dates (September 5, September 18, and November 30), the observed number of calls was statistically significantly greater than the model-predicted number of calls.

For observed versus predicted total daily 2-1-1 call volume before and after Hurricane Irma, the largest positive residuals based on the model were found during the hurricane and in the two to three weeks after the event. That is, the ARIMA model underestimated actual call volume, indicating a significant increase in 2-1-1 calls that appear temporally associated with Hurricane Irma. There were two days on which call volume exceeded the 99-percent CI in the first month after the hurricane, and one other day in December 2017 that does not appear temporally associated with a PHE.

Key Takeaways

This simple model fits the 2-1-1 Broward County call volume well, with the largest positive residuals seen during the hurricane and in the two to three weeks after the event. That is, the ARIMA model underestimated actual call volume, indicating a significant increase in 2-1-1 calls that appear temporally associated with Hurricane Irma. There were two days on which call volume exceeded the 99-percent CI in the first month after the hurricane, and there is one other day in December 2017 that does not appear temporally associated with a PHE.

We conducted a similar analysis using ARIMA modeling to examine 2-1-1 calls around the COVID-19 national emergency declaration (Analysis 11).

For observed versus predicted total daily 2-1-1 call volume before and after Hurricane Irma, the largest positive residuals based on the model were found during the hurricane and in the two to three weeks after the event. That is, the ARIMA model underestimated actual call volume, indicating a significant increase in 2-1-1 calls that appear temporally associated with Hurricane Irma. There were two days on which call volume exceeded the 99-percent CI in the first month after the hurricane, and one other day in December 2017 that does not appear temporally associated with a PHE.

Key Takeaways

We see a sharp increase in the residuals during the early weeks of the pandemic, with smaller isolated spikes occurring throughout the remainder of 2020 and the first half of 2021, indicating that total 2-1-1 call volume sharply increased early in the pandemic and intermittently spiked as the pandemic continued. There were some days when the call volume exceeded the 99-percent CI bounds for predicted calls, including seven days over the year before the beginning of the pandemic and four days in the two weeks immediately after the pandemic was declared. There were four other outliers over the following months.

Interrupted Time Series Analysis

Another modeling option to consider is ITSA, which allows you to determine whether PHEs are associated with a significant change in either the level of 2-1-1 calls per day or the slope or rate of change of 2-1-1 calls over time (Analysis 12).

The level of total 2-1-1 calls was almost 100 calls per day higher after Hurricane Irma, and the largest positive residuals occurred immediately after the disaster. The immediate increase in the level of calls after the disaster was relatively short-lived. Findings using the ITSA model suggest that although the hurricane appears associated with an increase in needs, this was not a sustained increase and the most acute increase in need for resources occurred in the immediate post-hurricane period.

Key Takeaways

The model output for the ITSA (not shown) is consistent with Analysis 12: The level of total 2-1-1 calls was almost 100 calls per day higher after Hurricane Irma, and the largest positive residuals occurred immediately after the disaster. The slope of calls (the change in the number of calls per day over time) is negative after the hurricane, suggesting that the immediate increase in the level of calls after the disaster was relatively short-lived. These findings suggest that although the hurricane appears associated with an increase in needs, this was not a sustained increase and the most acute increase in need for resources occurred in the immediate post-hurricane period.

Given the immediate, short-term increase in 2-1-1 calls after emergencies, PH agencies could communicate with 2-1-1 call centers to ensure that they plan in advance of PHEs for how to scale up 2-1-1 call capacity and/or community-based services.

We show similar figures for the onset of the COVID-19 pandemic in Analysis 13.

The level of total 2-1-1 calls was almost 84 calls per day higher after the declaration of the COVID-19 emergency, and the largest positive residuals occurred immediately after the start of the pandemic. The ITSA model yields a similar result after the emergency declaration, suggesting that the increase in the level of calls after the disaster appeared relatively sustained.

Key Takeaways

The model output for the ITSA (not shown) is consistent with Analysis 13: The level of total 2-1-1 calls was almost 84 calls per day higher after the declaration of the COVID-19 emergency, and the largest positive residuals occurred immediately after the start of the pandemic. The slope of calls (the change in the number of calls per day over time) is similar after the emergency declaration, suggesting that the increase in the level of calls after the disaster appeared relatively sustained.

Examples of Using 2-1-1 Data for Ongoing Monitoring

The previous sections of this exploratory analysis used ARIMA and ITSA to demonstrate that 2-1-1 calls appear sensitive to changes in BH after two types of PHEs and that they tend to remain elevated from baseline for about four weeks after an acute event, such as a hurricane, and eight weeks for the prolonged PHE of COVID-19. The figures demonstrate what it might look like to conduct ongoing monitoring of 2-1-1 calls in your jurisdiction and to assess, ideally in nearly real time, when this indicator exceeds a specified threshold.

In Analysis 14, we set an alert threshold at the 95th percentile of the distribution of seven-day moving averages of daily calls in the pre–Hurricane Irma period in our data (June to December 2016), depicted by the green dashed line at a value of 368 calls per day. This alert threshold results in eight alerts from June to December 2017, as the smoothed 2-1-1 call volume exceeded this threshold for eight days (September 19–26), which occurred about a week after the emergency declaration for Hurricane Irma ended.

2-1-1 calls increased during and after Hurricane Irma, peaking ten days after the end of the hurricane; during the peak, the seven-day moving average was approximately 108 calls above the moving average for the same date in 2016. This increase was not sustained, and 2017 call volume dropped back to 2016 levels by about early October 2017.

Key Takeaways

2-1-1 calls increased during and after Hurricane Irma, peaking ten days after the end of the hurricane; during the peak, the seven-day moving average was approximately 108 calls above the moving average for the same date in 2016. This increase was not sustained, and 2017 call volume dropped back to 2016 levels by about early October 2017.

This information could help PH agencies and their 2-1-1 call center partners anticipate the likely duration of increased community-level needs (and, therefore, stress) as signaled by 2-1-1 calls.

Analysis 15 shows a similar examination of the declaration of the national emergency for COVID-19.

2-1-1 calls increased after the declaration of the COVID-19 emergency, peaking about two weeks after the start of the disaster; during the peak, the seven-day moving average was approximately 108 calls above the moving average for the same date in 2019. This increase was not sustained, and 2020 call volume dropped back to 2019 levels by about late April 2020.

Key Takeaways

2-1-1 calls increased after the declaration of the COVID-19 emergency, peaking about two weeks after the start of the disaster; during the peak, the seven-day moving average was approximately 108 calls above the moving average for the same date in 2019. This increase was not sustained, and 2020 call volume dropped back to 2019 levels by about late April 2020.

As with the Hurricane Irma example, this information could help PH agencies and their 2-1-1 call center partners anticipate the likely duration of increased community stress after a PHE—and, therefore, how long additional BH education and resources may be needed.

Access selected publications that have used this data source for monitoring and surveillance.

Poison Control Center Calls

This exploratory analysis (the findings of which also appear in Disaster Medicine and Public Health Preparedness—see Fischer et al., 2023) demonstrates how you could monitor and assess for anomalies in trends in calls to PCCs for intentional exposures to potentially dangerous substances (hereafter called intentional exposures). We examined monthly intentional exposures in Dallas County, Texas, which was affected by both COVID-19, beginning in March 2020, and a severe winter storm, named Uri, in mid-February 2021. Uri caused widespread power outages during a period of unusually cold temperatures.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• For Dallas County during our period of interest, we had a sample size of 303,623 total calls, with a mean weekly call volume of 266 (range: 56 to 383).
• This sample included 5,281 intentional exposure calls, with a mean weekly intentional exposure call volume of 46 (range: 8 to 82). Of these intentional exposure calls, 3,880 were for suspected suicide, with a mean weekly call volume of 34 (range: 8 to 52).
• Age is missing for 5 percent of intentional exposure calls; when reported; the average age of callers is 30 years.
• Gender is missing for less than 1 percent of intentional exposure calls; 59 percent of calls are by women.
• Race/ethnicity information was not included for PCC calls in Dallas County.

Visualizing Trends in Intentional Exposures, Overall and by Subgroup

We explored trends in calls for intentional exposures, both overall and by gender and age group (i.e., adults versus youth). Analysis 16 shows a simple time series plot of the seven-day moving average of intentional exposure calls, including one year before and one year after the declaration of the COVID-19 pandemic as a national emergency. This plot allows us to examine overall trends, relationships to the key PHEs of interest (the pandemic and Winter Storm Uri), and any unusual values or potential outliers.

Calls for intentional exposures spiked in summer 2020 and again in late November, which coincided with local surges in COVID-19 cases (but not the declaration of the pandemic). We did not find an association between intentional exposures and Winter Storm Uri.

Key Takeaways

Calls for intentional exposures spiked in summer 2020 and again in late November, which coincided with local surges in COVID-19 cases (but not the declaration of the pandemic). We did not find an association between intentional exposures and Winter Storm Uri.

This analysis suggests that an indicator that may be sensitive to local severity of one type of PHE (e.g., a pandemic) may not be sensitive to another PHE type (e.g., a severe winter storm).

Analyses 17 and 18 show intentional exposure call volume stratified by gender (Analysis 17) and by age (Analysis 18) over the same period.

Intentional exposure call volume was consistently higher for women than for men. Calls for women spiked in early August 2020, whereas calls for men spiked in mid-November 2020, exceeding the seven-day moving average for women, which was not seen at other time points in our study data. These differences suggest not only more intentional exposure calls for women but also that the timing of spikes may differ by gender.

Key Takeaways

Intentional exposure call volume was consistently higher for women than for men. We see that intentional exposure calls for women spiked in early August 2020, whereas calls for men spiked in mid-November 2020, exceeding the seven-day moving average for women, which was not seen at other time points in our study data. These differences suggest that not only are there more intentional exposure calls for women but also that the timing of spikes may be different by gender.

The differential timing of spikes by gender could influence where and how PH agencies target their outreach activities related to suicide prevention.

Intentional exposure call volume was generally higher for adults than for youth. Intentional exposure calls for adults spiked in mid-September and mid-November, whereas intentional exposure calls for youth spiked in late October. Notably, the gap between the seven-day moving average of intentional calls for youth appears to trend upward beginning in the summer months of 2020, closing the gap between adults and youth that had been present prior to about October 2020.

Key Takeaways

We see that, when stratified by age, intentional exposure call volume was generally higher for adults than for youth. Intentional exposure calls for adults spiked in mid-September and mid-November, whereas intentional exposure calls for youth spiked in late October. Notably, the seven-day moving average of intentional calls for youth appears to trend upward beginning in the summer months of 2020, closing the gap between adults and youth that had been present prior to about October 2020.

These findings highlight the need for targeted PH interventions aimed at youth and could be used to justify requests for additional resources to address this crisis.

Assessing for a Significant Change After a Public Health Emergency

Interrupted Time Series Analysis

To assess whether the trends we noted when visualizing the data represented a statistically significant change in intentional exposure calls from the pre-PHE baseline, we used ITSA to examine whether PHEs (in this case, COVID-19 and Winter Storm Uri) were associated with a significant change in either the level of intentional exposure calls per day or the slope or rate of change of calls over time (Analysis 19).

The level and rate of change of intentional exposure calls were similar before and after the start of COVID-19 and Winter Storm Uri, but the level of intentional exposure calls per day was approximately two calls higher after the start of the summer surge in COVID-19 cases in Dallas County. The change in the number of calls per day over time is not significantly different after either PHE. These findings suggest that although the declaration of the COVID-19 emergency was not significantly associated with a change in intentional exposure calls in Dallas County, the surge in local cases in July and August 2020 appeared to temporally correspond with a spike in intentional exposure calls.

Key Takeaways

The model output for the segmented ITSA (not shown) is consistent with Analysis 19: The level and rate of change of intentional exposure calls were similar before and after the start of COVID-19 and Winter Storm Uri, but the level of intentional exposure calls per day was approximately two calls higher after the start of the summer surge in COVID-19 cases in Dallas County. The slope of calls—that is, the change in the number of calls per day over time—is not significantly different after either PHE. These findings suggest that although the declaration of the COVID-19 emergency was not significantly associated with a change in intentional exposure calls in Dallas County, the surge in local cases in July and August 2020 appeared to temporally correspond with a spike in intentional exposure calls.

This suggests that PHEs that affect a large geographic area, or even the entire country (e.g., COVID-19), may be associated with different BH impacts at different times, depending on the local severity of the crisis. This information may help PH agencies and their partners prioritize limited resources to prevent or address this BH impact.

Examples of Using Poison Control Center Calls for Intentional Exposures for Ongoing Monitoring

The previous section of this exploratory analysis used ITSA to demonstrate that PCC calls for intentional exposure appeared sensitive to changes in BH after the local surge in cases of COVID-19 in summer and fall of 2020 but not after Winter Storm Uri in February 2020. The figures that follow demonstrate what it might look like to conduct ongoing monitoring of intentional exposure calls in your jurisdiction and assess, ideally in nearly real time, when this indicator exceeds a specified threshold.

In Analysis 20, we set an alert threshold at the 95th percentile of the distribution of seven-day moving averages of daily calls in the pre-COVID-19 era in our data (March 2019–March 10, 2020), as depicted by the green dashed line, at a value of 7.9 calls per day. This alert threshold results in 128 alerts from March 11, 2020, to April 30, 2021, with the smoothed intentional exposure call volume exceeding this threshold for five consecutive days (March 15–19, 2020) immediately after the emergency was declared. Smoothed intentional call volume exceeded the threshold sporadically throughout the rest of the period spanning 2020 through April 2021.

Intentional exposure calls increased after the start of the COVID-19 pandemic, rising immediately after the national emergency declaration and then decreasing again until spiking in August, September, October, and mid-November 2020. Dallas County experienced its first peak of new reported COVID-19 cases in summer and early fall 2020 (corresponding roughly to the August and September peaks shown in Figure 20). Then, after a brief lull when new cases were low in the fall, rose dramatically again in November 2020, which was the start of a prolonged winter surge in Dallas. Therefore, there may be a temporal association between worsening local severity of the COVID-19 pandemic and intentional exposure calls.

Key Takeaways

Intentional exposure calls increased after the start of the COVID-19 pandemic, rising immediately after the national emergency declaration and then decreasing again until spiking in August, September, October, and mid-November 2020. Dallas County experienced its first peak of new reported COVID-19 cases in summer and early fall 2020 (corresponding roughly to the August and September peaks shown in Analysis 20). Then, after a brief lull when new cases were low in the fall, rose dramatically again in November 2020, which was the start of a prolonged winter surge in Dallas. Therefore, there may be a temporal association between worsening local severity of the COVID-19 pandemic and intentional exposure calls.

In response to this significant trend, PH agencies could target interventions that either prevent or address intentional exposures and deploy these interventions when agency monitoring shows the most need.

Access selected publications that have used this data source for monitoring and surveillance.

Prescription Medication Fills

For this exploratory analysis (the findings of which also appear in the Journal of Affective Disorders—see Levin, Acosta, and Faherty, 2022), we examined monthly prescription medication fills using the IQVIA Xponent dataset in two California counties (L.A. and Sacramento) affected by both COVID-19 and wildfires from March 2019 through June 2021. We focus on SSRIs to illustrate the broader concept of monitoring prescription medication fills in the PHE context, so we will refer to this data source as SSRI fills from here on in this section. These analyses show that you can detect a “signal” in the context of PHEs and how you could monitor (and assess for anomalies in) trends in SSRI fills.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• During our study period, we had an average monthly new SSRI fill rate of 986.7 per 100,000 residents in L.A. County (standard deviation [SD] = 52.0) and 889.4 per 100,000 residents in Sacramento County (SD = 40.9). Women accounted for 66.1 percent of all new fills in L.A. County and 68.5 percent of new fills in Sacramento County.
• Age distributions were similar across counties:
  • Ages 0–17 represented 6.7 percent of new fills in L.A. County and 7.6 percent of new fills in Sacramento County.
  • Ages 18–39 represented 34.1 percent of new fills in L.A. County and 32.7 percent of new fills in Sacramento County.
  • Ages 40–59 represented 29.7 percent of new fills in L.A. County and 32.1 percent of new fills in Sacramento County.
  • Ages 60+ represented 29.4 percent of new fills in L.A. County and 27.5 percent of new fills in Sacramento County.

Visualizing Trends in Prescription Medication Fills, Overall and by Subgroup

We first visualized the data from L.A. County using a simple time series plot of the counts of SSRI fills per 100,000 county residents (Analysis 21). This allowed us to look at overall trends, relationships to the key PHEs of interest, and potential outliers. The time series plot for Sacramento County looked similar (data not shown).

The start of the COVID-19 pandemic coincided with a decline in new SSRI fills in L.A. County, but the number per 100,000 residents rebounded by about June 2020. New SSRI fills then trended slightly upward in late 2020 and early 2021. A similar drop was not seen after the August 2020 wildfires. There appears to be some seasonality in the rate of new SSRI fills. The initial decline in new SSRI fills coincided with the stay-at-home orders at the onset of the COVID-19 pandemic, likely reflecting decreased health care access and by extension, fewer new prescriptions. The upward trend that began in late 2020 could suggest increasing incidence, intensity, or diagnoses of depression/anxiety, appearing temporally related to the start of COVID-19 and the wildfires in late summer 2020. There were no such stay-at-home orders after the wildfires; and it is possible that there was no change in health care access or new prescriptions.

Key Takeaways

The start of the COVID-19 pandemic coincided with a decline in new SSRI fills, but the number per 100,000 residents rebounded by about June 2020. New SSRI fills then trended slightly upward in late 2020 and early 2021. A similar drop was not seen after the August 2020 wildfires. There appears to be some seasonality in the rate of new SSRI fills. The initial decline in new SSRI fills coincided with the stay-at-home orders at the onset of the COVID-19 pandemic, likely reflecting decreased health care access and, by extension, fewer new prescriptions. The upward trend that began in late 2020 could suggest increasing incidence, intensity, or diagnoses of depression and/or anxiety, appearing temporally related to the start of COVID-19 and the wildfires in late summer 2020. There were no such stay-at-home orders after the wildfires, and it is possible that there was no change in health care access or new prescriptions.

We next explored whether these trends in SSRI fill rates for L.A. and Sacramento counties varied when stratified by gender and by age (Analyses 22 and 23, respectively).

Women had approximately twice the SSRI fill rate of men throughout the entire study period. After the onset of COVID-19 and the wildfires, the SSRI fill rate for women appeared to increase more quickly than for men, slightly widening the gap in SSRI fills by gender.

Fill rates for children under 17 were substantially lower than for other age groups, which is to be expected because of guidelines around pediatric prescribing of SSRIs. The increasing trend in fill rates over time appears to be driven by a more pronounced increase among those 18 to 39 years old.

Key Takeaways

Analysis 22 shows that women had approximately twice the SSRI fill rate of men throughout the entire study period. After the onset of COVID-19 and the wildfires, the SSRI fill rate for women appeared to increase more quickly than for men, slightly widening the gap in SSRI fills by gender.

Analysis 23 shows that fill rates for children under 17 were substantially lower than for other age groups, which is to be expected because of guidelines around pediatric prescribing of SSRIs. The increasing trend in fill rates over time appears to be driven by a more pronounced increase among those 18 to 39 years old.

By identifying that overall increases in new SSRI fills around the time of the COVID-19 pandemic's onset and wildfires may be driven by women and by those 18–39 years old, PH officials could design targeted interventions for these subpopulations.

Assessing for a Significant Change After a Public Health Emergency

We used ITSA with robust standard errors to estimate whether the level and slope (rate of change) of new SSRI fill rates for L.A. and Sacramento counties showed statistically significant changes after the PHEs of interest compared with the pre-PHE period (March 2019–February 2020). An ITSA can be a useful statistical method in the absence of a control county unaffected by the PHE. We visualize the results of the ITSA model in Analysis 24, with observed monthly data shown in blue and the fitted lines, based on the regression coefficients, shown in green. We excluded March 2020 from the models because the pandemic began in the middle of that month; thus March 2020 cannot clearly be classified as before or after the start of the PHE with our monthly data.

There were no statistically significant changes in the outcome levels immediately after the start of COVID-19 (p > 0.05) or after the start of the wildfires (p > 0.05). In L.A. County, SSRI fills rose gradually after the onset of COVID-19, but fills in September 2020 after the wildfires were not significantly different from the first few months of COVID-19 (p > 0.05).

In Sacramento County, SSRI fills following the onset of COVID-19 or the wildfires were not significantly different from the pre-COVID-19 or the pre-wildfire numbers.

Key Takeaways

The model output for the ITSA (not shown) confirms what you can see in Analysis 24: For both panels, there were no statistically significant changes in the outcome levels immediately after the start of COVID-19 (p > 0.05) or after the start of the wildfires (p > 0.05). In panel A (i.e., L.A. County), the slope of the trend line following the onset of COVID-19 is significantly greater than the pre-COVID-19 slope (p < 0.01), but the slope after the wildfires in September 2020 is not significantly different from the slope during the first few months of COVID-19 (p > 0.05). In panel B (i.e., Sacramento County), the slopes following the onset of COVID-19 or the wildfires were not significantly different from the pre-COVID-19 or pre-wildfire slopes (p > 0.05).

These results suggest that the upward trend in new SSRI fills in L.A. County after COVID-19 is probably not due to chance alone. In response to this significant trend, PH agencies could consider various interventions to improve overall mental health and access to mental health treatment. These activities may include scaling up depression screenings; mental health services; and distribution of promotional materials on treatment locations, stigma reduction, and suicide prevention.

Controlling for Seasonality to Detect Anomalies and Significant Changes

In Analysis 25, we plot the number of new SSRI fills per 100,000 residents in L.A. County by month, superimposing lines for the years 2019, 2020, and 2021, to account for seasonality of SSRI fills. We then extend the ITSA model above, adding time fixed effects (specifically, coefficients for each calendar month and calendar year) to control for differences in the outcomes that are associated with these terms but unrelated to the PHE (output not shown but summarized in the Key Takeaways paragraph).

SSRI fills have similar peaks in certain months regardless of year and timing of PHEs (e.g., March, July, October). SSRI fills in April and May 2020 were lower than the same months in 2019 but fills for the rest of 2020 were consistently higher than the same months in 2019. Fills during the first half of 2021 exceeded those during the same months in 2019 and 2020, suggesting that SSRI fill rates in L.A. County have trended up since about May 2020.The model output for the ITSA with time fixed effects confirmed these visual takeaways: Controlling for seasonal trends, there was a statistically significant decrease in new SSRI fills immediately after the start of COVID-19 (p < 0.01) and a significantly greater slope of the trendline following the onset of COVID-19 compared with the pre-COVID-19 slope (p < 0.05). The outcome levels and slopes from before and after the wildfires did not significantly differ from each other (p > 0.05).

Key Takeaways

Analysis 25 confirms the seasonality of SSRI fills, with similar peaks seen in certain months regardless of year and timing of PHEs (e.g., March, July, October). SSRI fills in April and May 2020 were lower than in the same months in 2019, but fills for the rest of 2020 were consistently higher than in the same months in 2019. Fills during the first half of 2021 exceeded those during the same months in 2019 and 2020, suggesting that SSRI fill rates in L.A. County have trended upward since about May 2020. The model output for the ITSA with time fixed effects confirmed these visual takeaways: Controlling for seasonal trends, there was a statistically significant decrease in new SSRI fills immediately after the start of COVID-19 (p < 0.01), and there was a significantly greater slope of the trend line following the onset of COVID-19 compared with the pre-COVID-19 slope (p < 0.05). The outcome levels and slopes from before and after the wildfires did not significantly differ from each other (p > 0.05).

These results confirm that monitoring BH at a population level must account for seasonality of SSRI fills to detect spikes that are more likely to reflect true increases than to reflect seasonal changes or individuals who ran out of refills after a three-month period.

In Analysis 26, we plot the ratio of the number of new SSRI fills in L.A. County by month after the declaration of the COVID-19 national emergency to the number of new fills for the same month before the onset of COVID-19. We then added several red lines that represent two different thresholds of change relative to the pre-COVID-19 months. These thresholds are meant to serve as examples and do not necessarily represent population-level anomalies.

In May 2020, the ratio of post-COVID-19 SSRI fills to pre-COVID-19 fills declined by more than 5 percent, but beginning in June 2020, the ratio consistently exceeded 1.0, indicating that the fill rate was higher than it was in the same month pre-pandemic. For many months after the declaration of the COVID-19 national emergency, SSRI fill rates showed an increase of more than 5 percent compared with the same month pre-pandemic, and in March 2021 and June 2021, we observe increases of more than 10 percent compared with the same pre-pandemic month

Key Takeaways

In May 2020, the ratio of post-COVID-19 fills to pre-COVID-19 fills (i.e., blue line) declined by more than 5 percent, but beginning in June 2020, the ratio consistently exceeded 1.0, indicating that the fill rate was higher than it was in the same month pre-pandemic. For many months after the declaration of the COVID-19 national emergency, SSRI fill rates showed an increase of more than 5 percent compared with the same month pre-pandemic, and in March 2021 and June 2021, we observe increases of more than 10 percent compared with the same pre-pandemic months.

This figure provides a framework for using quantitative thresholds to identify potential anomalies that are adjusted for seasonality. Selecting the exact quantitative threshold for determining an anomaly (e.g., 5 percent, 10 percent) is subjective and will vary by municipality capacity, based on population, to respond to alerts, as well as other factors.

Access selected publications that have used this data source for monitoring and surveillance.

Over-the-Counter Sleep Aid Sales

This exploratory analysis examines how you could monitor (and assess for) anomalies in OTC sleep aid sales surrounding PHEs. We examined sales from four OTC products intended to improve sleep. The active ingredients in these products included melatonin, herbal remedies, and diphenhydramine (when marketed as a sleep aid instead of allergy relief). We examined daily data from a retail chain in L.A. County, covering a period that included the COVID-19 pandemic and the wildfires in August 2020.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• Each data point represents aggregated sales data across all stores of a retail chain in L.A. County from January 1, 2017, to June 13, 2022. There are 14 missing values, resulting in a total of 1,976 samples.
• The average sales volume per day for OTC sleep aids was approximately 12 products. The average sales volume across all monitored categories was approximately 6,500. On average, the four OTC sleep aid products we examined composed only a small fraction of sales, between 0.001 percent and 0.1 percent of total sales across 20 product categories.
• We defined the onset of the COVID-19 pandemic as March 20, 2022, and the onset of wildfire season as August 15, 2022.
• Demographic information on the people purchasing these products is not available in these data, as each observation is a sale of a particular OTC product.

Visualizing Overall Trends in Over-the-Counter Sleep Aid Sales

We explored overall trends in OTC sleep aid sales. Analysis 27 shows a time series plot of the seven-day lagged moving average of OTC sleep aid sales between 2017 and 2022. This plot allows us to examine overall trends in the data and identify whether the PHEs of interest are associated with changes in OTC sleep aid sales. Analysis 28 shows the same data, plotted to enable comparisons of the same calendar period across different years.

OTC sleep aid sales in L.A. County were climbing gradually from 2017 until the start of the pandemic, at which point sales began a dramatic increase. Sales remained constant at the new high level directly after the wildfires, dropping off from November 2020 until January 2021. Sales spiked in March 2021 then declined again.

Key Takeaways

The COVID-19 pandemic was associated with a steep increase in OTC sleep aid sales in L.A. County, with volume increasing by a factor of 15 between the low in March 2020 and the peak in September 2020. After an early surge in OTC sleep aid sales just after the pandemic declaration, sales continued to increase, initially peaking in August 2020. The onset of wildfires in L.A. County appears to coincide with a brief decrease in OTC sleep aid sales, but sales quickly rebounded. Although sales declined after their second peak in March 2021 (notably, the first anniversary of the pandemic declaration), OTC sleep aid sales remain much higher than pre-pandemic levels in 2022.

This information may be useful for PH agencies in informing public awareness campaigns about the risk of unintentional ingestions of melatonin, for instance, or about lifestyle modification (e.g., non–medication-related) strategies to address disrupted sleep. Because sleep disruptions may be a symptom of more-severe mood disturbances, PH agencies could also triangulate these findings with other data sources to understand the magnitude of the problem of BH impacts in their jurisdiction of a recent or ongoing PHE.

Pre-pandemic, OTC sleep aid sales volumes were similar each year in L.A. County, and there were slight increases in sales during the summer months. After the start of the pandemic, sales increased dramatically through late 2020, trended down from November 2020 through January 2021, then increased to their highest level around the time of the first anniversary of the pandemic declaration (March 2021), before steadily declining through the rest of 2021. OTC sleep aid sales appear to have plateaued in early 2022.

Key Takeaways

Plotting OTC sleep aid sales from different years on the same x-axis shows that 2020 and 2021 were anomalous relative to the pre-pandemic period. Before the pandemic, volumes were similar each year, and there were slight increases in OTC sleep aid sales during the summer months. After the start of the pandemic, sales increased dramatically through late 2020, trended downward from November 2020 through January 2021, then (as also shown in Analysis 28) increased to their highest level around the time of the first anniversary of the pandemic declaration (March 2021), before steadily declining through the rest of 2021. OTC sleep aid sales appear to have plateaued in early 2022 (not shown).

These data suggest that ongoing PHEs can create dynamic and long-term changes in health behaviors. Insomnia is often caused by stress and anxiety, so these results may suggest an immediate and sustained need for increases in access to mental health services; depression screenings; and the distribution of promotional materials on treatment locations, stigma reduction, and suicide prevention.

Assessing for a Significant Change After a Public Health Emergency

To assess whether PHEs were associated with short- or long-term changes in normalized OTC sleep aid sales, we conducted two analyses: an ITSA and a change-point analysis (CPA).

Interrupted Time Series Analysis

ITSA identifies whether specified events were associated with a change in either the level or the slope of sleep aid sales (Analysis 29).

The onset of the COVID-19 pandemic was associated with a large increase in sleep aid sales in L.A. County, suggesting a sustained increase in the use of sleep aids. The wildfires findings could be confounded by changing pandemic conditions.

Key Takeaways

The onset of the COVID-19 pandemic was associated with a large slope increase in sleep aid sales, suggesting a sustained increase in the use of sleep aids. The wildfires are associated with a small level increase and a slope decrease, but this finding could be confounded by changing pandemic conditions.

Change-Point Analysis

A CPA identifies when the mean of a time series may have changed (see Analysis 30). Importantly, it does not require the date of an event as an input, so if a change point coincides with a known event, it provides more-compelling evidence that the event was linked with the change.

A change point analysis used two change points: The first occurred during the initial COVID-19 surge in L.A. County in May 2020, and indicates normalized OTC sleep aid sales remained elevated for 18 months (but did not change again around the time of the wildfires of 2020). The second change point occurred in September 2021: This change does not coincide with a known event but instead may signal the establishment of a new baseline as the population adjusted to life with COVID-19. This new, lower mean of OTC sleep aid sales is still more than twice the pre-pandemic mean, suggesting that the increase in the use of sleep aids associated with COVID-19 is persisting in L.A. County.

Key Takeaways

This CPA used two change points. The first occurred during the initial COVID-19 surge in L.A. County in May 2020, and mean normalized OTC sleep aid sales remained elevated for 18 months (but did not change again around the time of the wildfires of 2020). The second occurred in September 2021: This change does not coincide with a known event but instead may signal the establishment of a new baseline as the population adjusted to life with COVID-19. This new, lower mean of OTC sleep aid sales is still more than twice the pre-pandemic mean, suggesting that the increase in the use of sleep aids associated with COVID-19 is persisting in L.A. County.

This information could be useful to PH agencies and their partners as they plan ongoing pandemic recovery efforts.

Examples of Using OTC Sleep Aid Sales Data for Ongoing Monitoring

The two previous analyses showed that COVID-19 was associated with large, sustained increases in normalized OTC sleep aid sales, even two years after the pandemic was declared. In contrast, the analyses in this section rely on data collected over periods both before and after the event of interest. Therefore, they have limited value for ongoing monitoring of a PHE as it evolves (i.e., identifying whether real-time measurements are anomalous and warrant further investigation).

In contrast, in the next section and Analysis 31, we illustrate a three-step method for continual monitoring:

• First, build a predictive time series model based on a period of normal activity (in this case, pre-pandemic). Here, we constructed an ARIMA model.
• Second, use this model to make predictions and look at the differences between the prediction and the observed value (known as residuals).
• Finally, create alerts based on these residuals.

ARIMA Analysis with Alerts

We built a basic ARIMA model with a single level of differencing and one moving average term, based on OTC sleep aid sales data from 2017 and 2018. We then applied this model to the entire time series (shown in green in Analysis 31) and calculated residuals (the distance between the green line and the purple line on a given day). Finally, we created three different alert rules, described in more detail in Part 3.

• The Shewhart alert triggers if the value of the residual is much larger than expected.
• The cumulative sum (CUSUM) alert triggers if the sum of the last few residuals is much larger than expected.
• The EWMA alert triggers if a weighted average of the residuals is larger than expected (more-recent residuals are weighted higher).

The Shewhart is best at detecting brief “spikes”, whereas the other methods may be more appropriate for long, slow increases in the indicator of interest.

Shewhart, CUSUM, and EWMA alerts were triggered at similar times, indicating approximately four periods of anomalous activity in OTC sleep aid sales in L.A. County. These roughly coincided with the period just after each of four large COVID-19 waves in L.A. County. During the start of each COVID-19 wave, sleep aid sales appeared to dip slightly, perhaps because people were purchasing relatively more OTC medications to self-treat their COVID-19 symptoms (e.g., fever, cough). Sleep aid sales then increased after the peak of the COVID-19 wave passed.

Key Takeaways

The three alert types were triggered at similar times. We see approximately four periods of anomalous activity in OTC sleep aid sales. These roughly coincided with the period just after each of four large COVID-19 waves in Los Angeles County. During the start of each COVID-19 wave, sleep aid sales appeared to dip slightly, perhaps because people were purchasing relatively more OTC medications to self-treat their COVID-19 symptoms (e.g., fever, cough). Sleep aid sales then increased after the peak of the COVID-19 wave passed. A potential explanation for this trend, which warrants further exploration, is that sleep aid sales increased as cases subsided due to the population coping with isolation, disrupted routines, lingering symptoms of COVID-19 (of note, long COVID-19 is increasingly being linked to sleep disturbances [Bhat and Chokroverty, 2022]), or simply the stress of the ongoing pandemic.

Access selected publications that have used this data source for monitoring and surveillance.

Emergency Department Visits

Explore an overview of this data source, including how to access it and its strengths and limitations.

There exist numerous PH applications of using the NSSP for BH surveillance. For example, the Washington State DOH has a unique syndromic surveillance system, the Rapid Health Information Network (RHINO). As part of the NSSP, ED data from across the state are uploaded to RHINO and forwarded to NSSP for inclusion in the BioSense Platform and related national syndromic surveillance activities.

During the COVID-19 pandemic, the Washington State DOH conducted “syndromic surveillance on five behavioral health conditions that can be detected in emergency department settings: psychological distress, suicidal ideation, suspected suicide attempts, alcohol-related conditions, and suspected all-drug overdoses” (NSSP, 2021b). These analyses and related insights were shared in weekly situation reports and forecasts, providing PH leadership and practitioners across the state with an understanding of BH needs and crises.

These situation reports, for instance, provided longitudinal trends on the total and relative counts of statewide ED visits for psychological distress, SI, and suspected suicide attempts. Analysis 32 illustrates trends in the relative count of ED visits for SI among youth in Washington by week from 2019 to 2021.

The relative count of ED visits for SI among youth in Washington in weeks 20-40 for 2019 hovered at around 600; that number was closer to 900 for the same weeks in 2020. Weeks 0-20 in 2021 were the highest, closer to 1,000 which is significantly higher than the same weeks in 2019 or 2020.

Source: Reprodcued from Washington State DOH, 2020; Georgoulas-Sherry and Franzen, 2021.

Using these weekly updates and trends for psychological distress, SI, and suicide attempts, the Washington State DOH and its partners (e.g., BH response teams) “developed a targeted public health response that included issuing provider alerts (e.g., suicide and overdose risk) and creating county-specific maps to guide local health jurisdictions’ response efforts” (NSSP, 2021). These efforts also encompassed improving education and awareness, such as a Coping with COVID blog series (Washington State DOH, undated-b) and podcast (Washington State DOH, undated-a) for residents of the state, with such topics as “It’s Not Just You,” “Regulating Emotions,” “Exhausted Families,” and “Developing Resilience.”

Other localities at the state and local levels are also leveraging the NSSP in similarly important ways. The Indiana DOH (undated) developed an Overdose Spike Response Toolkit that recommends identifying and responding to overdose spikes detected through syndromic surveillance. Whenever there is a spike or an anomaly in ED visits for overdoses across the state, an alert is sent to the appropriate local health department, which can then respond in several ways:

• conduct further surveillance of the spike to confirm it, using multiple data sources (e.g., 911 calls, drug overdose deaths reported by coroners, increase in rates of EMS treating suspected overdoses and reversing them with naloxone, ED visits for overdose symptoms, PCC calls, police reports of illicit drug arrests, and toxicology testing)
• communicate with local partners about the spike
• mobilize harm reduction strategies, such as naloxone distribution, community messaging, drug disposal and safe storage programming, efforts to promote linkage to care for substance use disorders, and syringe exchange programs (Indiana DOH, undated).

A third example of syndromic surveillance in action, shown in Analysis 33, is from the Tri-County Health Department in Colorado, which shows SI (brown), suicide attempts (blue), and alcohol overdoses (dark green) among youth from January 1, 2020, through January 1, 2022. PH guidelines and orders are shown in light green vertical bars.

ED visits among youth for SI and suicide attempts are higher than those for alcohol overdose from January 1, 2020, through January 1, 2022 in Colorado counties. ED visits for SI are the highest of the three peaking at over 30 per week compared to alcohol overdose, which peaks at five per week.

Source: Reproduced from Colorado Tri-County Health Department, 2022.

For more examples of states using syndromic surveillance data to monitor BH, see

• "Idaho Uses Syndromic Data to Help Understand Who Is at Risk for Suicide" (NSSP, 2022a)
• "Louisiana Takes Action Against Drug Abuse by Sharing Syndromic Data" (NSSP, 2022b)
• "State of Washington Uses Suicide-Related Syndromic Surveillance to Guide Behavioral Health Response During COVID-19 Pandemic" (NSSP, 2022c)
• "Syndromic Data and Collaboration Enhance Substance Overdose Surveillance" (NSSP, 2022d).

Access selected publications that have used this data source for monitoring and surveillance.

EMS Activations

Here, we provide examples of how NEMSIS data have been used to detect a BH-related “signal” in the context of PHEs and to monitor (and assess for anomalies in) trends in BH-related EMS activations.

Explore an overview of this data source, including how to access it and its strengths and limitations.

Describing the Data Used in This Exploratory Analysis

Characteristics of the Data

• The total numbers of EMS activations per year across the entire United States were
  • 2018: 16,919,498
  • 2019: 22,800,494
  • 2020: 28,291,393
  • 2021: 31,729,895
  • 2022: 26,632,503
• Analysis 34 illustrates the variation in sample size of EMS activations within and across years for 2018–2022.

Around the time of the COVID-19 national emergency declaration on March 11, 2020, the number of EMS activations decreased by approximately 34 percent between week 10 (March 2 to March 8, 2020) and week 17 (April 20 through April 26, 2020).

Source: Reproduced from Mann, 2023.

Key Takeaways

Around the time of the COVID-19 national emergency declaration on March 11, 2020, the number of EMS activations decreased by approximately 34 percent between week 10 (March 2 to March 8, 2020) and week 17 (April 20 through April 26, 2020). The number of EMS activations began a second downward trend in week 28 of 2020.

Examining overall patterns in total EMS activations is important for interpreting trends in suicide-related activations, discussed in the next section.

Visualizing Trends in Behavioral Health–Related EMS Activations

A simple time series plot (Analysis 35) shows the count of 9-1-1–initiated EMS activations related to suicide (y-axis) across weeks of the year (x-axis) over multiple years (different-colored lines). This figure allows us to examine overall trends in EMS activations related to suicide, relationships to the key PHE of interest (COVID-19), and any unusual values or potential outliers.

Overall, there were more EMS activations related to suicide or self-harm in 2020 and 2021 than in 2018 and 2019. There was a sharp decrease in the count of suicide-related EMS activations in the weeks following the declaration of the COVID-19 national emergency. Counts gradually returned to the pre-COVID-19 baseline for 2020 before again trending downward in the last few months of that year. The count of suicide-related EMS activations appears to trend downward beginning in August 2021 until the most recent data available (October 2021).

Source: Reproduced from NEMSIS, 2023.

Key Takeaways

Overall, there were more EMS activations related to suicide or self-harm in 2020, 2021, and 2022 than in 2018 and 2019. There was a sharp decrease in the count of suicide-related EMS activations in the weeks following the declaration of the COVID-19 national emergency. Counts gradually returned to the pre-COVID-19 baseline for 2020 before again trending downward in the last few months of that year. The count of suicide-related EMS activations appears to trend downward beginning in May 2022.

This information may be useful for planning and resource allocation.

Importantly, the apparent decrease in counts of suicide-related activations shown in Analysis 36 may not be due to a decrease in actual suicide attempts or deaths by suicide, but may be influenced by

• decreases in overall EMS activations reported
• increases in the number of states submitting data
• changes in the number of EMS agencies in each state reporting data.

To better understand the trends in counts, we can look at the rate of suicide-related EMS activations (i.e., the percentage of total EMS activations that were related to suicide) for the same period, from 2018 to 2022 (Analysis 36).

The percentage of EMS activations related to suicide was relatively similar year over year but with spikes in certain weeks of certain years.

Source: Reproduced from Mann, 2023.

Key Takeaways

Analysis 36 shows that the percentage of EMS activations related to suicide was relatively similar year over year but with spikes in certain weeks of certain years.

These spikes are examples of signals in your data that you may want to investigate further to decide whether and how to respond.

Assessing for a Significant Change After a Public Health Emergency

Recent studies (Glober et al., 2020; Khoury et al., 2021) provide details on how researchers have applied ARIMA and period-over-period (similar to DiD) analytic techniques to examine whether there was a statistically significant change in BH-related EMS activations from a pre-PHE baseline.

For example, Analysis 37 illustrates findings from Glober et al. (2020), who analyzed daily EMS activations in Marion County, Indiana (where Indianapolis is located), in which naloxone was administered (Panel A), daily rates of all overdoses (Panel B), daily rates of all EMS calls (Panel C), and drug overdoses per week (Panel D). Blue vertical lines indicate the stay-at-home order date (March 25, 2020) and the beginning of reopening in Indianapolis (May 16, 2020). Red-shaded regions indicate the 99-percent CI for the ARIMA forecast over the period after the stay-at-home order (estimated using data up through March 24, 2020).

Panel A

Panel A: An analysis of daily EMS activations in Marion County, Indiana (where Indianapolis is located) in which Naloxone was administered , daily rates of all overdoses, daily rates of all EMS calls, and drug overdoses per week show consistent increases in drug overdoses beginning with the stay-at-home order being issued and a continued increase after the stay-at-home order was terminated. Total EMS activations increased 4 percent; overdose activations increased 43 percent; activations where naloxone was administration increased 61 percent; and deaths from drug overdoses increased 47 percent over the same period (i.e., after the stay-at-home order).

Panel B

Panel B: An analysis of daily EMS activations in Marion County, Indiana (where Indianapolis is located) in which Naloxone was administered , daily rates of all overdoses, daily rates of all EMS calls, and drug overdoses per week show consistent increases in drug overdoses beginning with the stay-at-home order being issued and a continued increase after the stay-at-home order was terminated. Total EMS activations increased 4 percent; overdose activations increased 43 percent; activations where naloxone was administration increased 61 percent; and deaths from drug overdoses increased 47 percent over the same period (i.e., after the stay-at-home order).

Panel C

Panel C: An analysis of daily EMS activations in Marion County, Indiana (where Indianapolis is located) in which Naloxone was administered, daily rates of all overdoses, daily rates of all EMS calls, and drug overdoses per week show consistent increases in drug overdoses beginning with the stay-at-home order being issued and a continued increase after the stay-at-home order was terminated. Total EMS activations increased 4 percent; overdose activations increased 43 percent; activations where naloxone was administration increased 61 percent; and deaths from drug overdoses increased 47 percent over the same period (i.e., after the stay-at-home order).

Panel D

Panel D: An analysis of daily EMS activations in Marion County, Indiana (where Indianapolis is located) in which Naloxone was administered, daily rates of all overdoses, daily rates of all EMS calls, and drug overdoses per week show consistent increases in drug overdoses beginning with the stay-at-home order being issued and a continued increase after the stay-at-home order was terminated. Total EMS activations increased 4 percent; overdose activations increased 43 percent; activations where naloxone was administration increased 61 percent; and deaths from drug overdoses increased 47 percent over the same period (i.e., after the stay-at-home order).

Source: Reproduced from Glober et al., 2020.

Key Takeaways

These four panels show consistent increases in drug overdoses beginning with the stay-at-home order being issued and a continued increase after the stay-at-home order was terminated. Total EMS activations increased 4 percent; overdose activations increased 43 percent; activations where naloxone was administered increased 61 percent; and deaths from drug overdoses increased 47 percent over the same period (i.e., after the stay-at-home order).

These spikes suggest that PH agencies could consider prioritizing overdose prevention efforts in advance of (or in conjunction with) future stay-at-home orders and that these prevention efforts should continue beyond the lifting of those stay-at-home orders. These efforts can include expanding availability of naloxone, distributing promotional materials on treatment services and stigma reduction, and increasing the number of safe opioid disposal collection sites.

Access selected publications that have used this data source for monitoring and surveillance.