F: Additional Detail on Advanced Analytic Methods

Anomaly Detection

Table 1 includes two potential approaches to anomaly detection.

Examples of Approaches to Anomaly Detection

Approach	Description
Shewhart algorithm	This algorithm triggers an alert when yt, the BH indicator count or level on a single day t, exceeds a threshold h. General guidance is to set the alert threshold to yield a false positive rate of 1 percent. You get an alert if the value of the residual is much larger than expected. This algorithm is best at detecting brief “spikes,” while others may be more appropriate for more-gradual increases in the BH indicator of interest.
CUSUM statistic	The CUSUM statistic cumulates the residuals, which can be plotted against time. The CUSUM control chart’s parameter k is called the reference value, or the allowance, and is related to the size of the smallest shift that you want to detect. Deviations smaller than k are ignored. You get an alert if the sum of the last few residuals is much larger than expected.

You can also define your own alert threshold. You will want to assess or monitor the performance of the algorithms shown in Table 1. Performance is usually measured by the sensitivity, specificity, and timeliness of the alert (Cowling et al., 2006):

• Sensitivity (or true positive) implies that the alarm “goes off” when there is an actual shift in the BH indicator after a PHE. For example, this may be measured as sounding at least one alert during the PHE period. False positive implies sounding the alert when there is not an actual shift after a PHE.
• Specificity is equal to (1 − false positive rate). Some have defined specificity as (1 − r/n), where r is the number of alerts outside the peak (PHE) period (or false alarms) and n is the number of periods outside the peak (PHE) period.
• Timeliness (or lag) is defined as the length of time (days, weeks) between the beginning of the PHE and the first time that an alarm is raised.

The choice of alert threshold requires balancing sensitivity and specificity considerations. Each alert is usually followed by manual review of the data; you will need to consider your department’s capacity for this task. It may also be helpful to set thresholds for both activating and deactivating a PH response.

Additionally, you may want to weigh the implications of not setting off the alert in a timely fashion—where timeliness itself depends on the BH indicator and expected lag in the BH impact of a PHE. Small shifts or slow increases in cases related to BH may need a longer lag or period to be detected by the proposed algorithms; slow outbreaks also may not be detected with the same accuracy as fast outbreaks of infectious diseases (Stoto, Schonlau, and Mariano, 2004). Finally, a statistic that accumulates deviations, such as the CUSUM, will likely be more able to detect a slow but steady change in a BH indicator related to mental health.

For examples of analyses using these different methods for anomaly detection, click on the buttons.

Access a sample analysis showing the use of a moving average to set an alert threshold for 2-1-1 call volume in Broward County, Florida, after a hurricane and the start of the COVID-19 pandemic. Learn about a comparison of Shewhart, CUSUM, and EWMA alerts for assessing changes in sleep aid sales in L.A. County after the start of the COVID-19 pandemic and wildfires. Access a sample analysis showing the use of different alert thresholds for assessing changes in new SSRI fills in L.A. County after COVID-19.

Advanced Methods for Statistical Testing

If you have access to advanced data science capacity, you may want to pursue more-complex analyses, such as

• ITSA, which uses repeated assessments to determine whether a PHE “interrupted” an existing trend
• CPA, which identifies a point or points in a time series when a change or break in the trend took place
• DiD analysis, which compares a group that has experienced a PHE with one that has not to estimate a causal effect.

To use these methods, you will need data from periods both before and after the PHE. The purpose of these methods is different from real-time monitoring and surveillance after an event. These methods help you understand the BH impacts of a past PHE in your jurisdiction, which could allow you to better prepare for the next event. In addition, these analyses may tell you that the population is still experiencing impacts from the prior event.

Each method is described in more detail in the following sections.

Interrupted Time Series Analysis

In an ITSA, repeated assessments of a BH indicator are assumed to be interrupted by a PHE. The hypothetical scenario under which the PHE has not occurred and the trend in the indicator of interest continues unchanged is called the counterfactual. This counterfactual scenario provides a comparison for the evaluation of the potential impact of the PHE by examining the change in the indicator after the PHE. However, the absence of a true control or comparison group means that there may be another reason besides the PHE for the observed interruption. Thus, you cannot make causal claims about the relationship between the PHE and the BH indicator.

Access sample code in STATA to generate an ITSA plot with your data. Access a sample ITSA using PCC calls related to intentional ingestions in Dallas County, Texas, during COVID-19.

Change-Point Analysis

While a time series plot may indicate an upward trend in the BH indicator of interest, if you want to identify the specific point at which a significant change occurred, you could use CPA. Compared with ITSA, this approach is more robust to outliers in your data and is better at detecting smaller, sustained shifts in the indicator you are monitoring. It also better characterizes such shifts. As demonstrated in our exploratory analysis of OTC sleep aid sales, you can identify multiple change points over time, and you can interpret associated confidence levels, with higher levels (closer to 100 percent) indicating greater certainty that a statistically significant change took place.

Access sample code in R to generate an ARIMA plot with your data. Access a sample CPA of OTC sleep aid sales in L.A. County during COVID-19 and the wildfires of 2020.

Difference-in-Differences Analysis

The DiD analysis method is a quasi-experimental approach for observational studies. For this method, you will need data from a population that has been exposed to a PHE and from a population that has not been exposed—or is not experiencing (or did not experience) the PHE. The other two methods already described (i.e., single-group ITSA and CPA) do not have a comparable control group, nor do they control for secular trends that may induce change simultaneously; therefore, the pre-event trend projected into the post-event period serves as the counterfactual. In contrast, for DiD analysis, you need this unexposed group to obtain an appropriate counterfactual for estimating a causal effect of the PHE. The use of a well-matched control group will yield more-robust evidence of the effect of a PHE on the BH indicator(s) of interest. In our exploratory DiD analysis using UI claims data from L.A. County, we used a fixed effects approach to “difference out” both place effects that are constant over time and time trends that are constant over place. This approach incorporates how place may affect the level of the BH series and accounts for natural variation across time, such as seasonality, in the time series.

DiD can be a form of an ITSA or a CPA, depending on how you parameterize (i.e., set up) your DiD model. You can evaluate an interruption in your data (as in an ITSA) or identify change points and their durations (as in a CPA).

Access a sample DiD analysis of UI claims in L.A. County during COVID-19 and the wildfires of 2020.