The authors use simulations to assess the performance of a wide range of statistical models commonly used in the gun policy literature to estimate the effects of state-level gun policies on firearm deaths and to identify the most-appropriate statistical methods for producing estimates. The results suggest substantial statistical problems with many of the methods used in this field. The authors identify the best method among those assessed.

Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths

A Simulation Study

Terry L. Schell, Beth Ann Griffin, Andrew R. Morral

ResearchPublished Dec 10, 2018

Cover: Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths
Download PDF
Additional Downloads

Includes Data Files

The RAND Corporation launched its Gun Policy in America initiative with the goal of creating objective, factual resources for policymakers and the public on the effects of gun laws. As a part of this project, RAND researchers conducted a systematic literature review and evaluation of scientific studies on the effects of 13 classes of policies. One of the findings of the review was that the effects of policies estimated in the literature appeared to be sensitive to the specific statistical methods that were employed. This suggests the importance of identifying the most-appropriate statistical methods to use on these data.

In this report, the authors use simulations to assess the performance of a wide range of statistical models commonly used in the gun policy literature to estimate the effects of state-level gun policies on firearm deaths. The study aimed to identify the most-appropriate statistical modeling and analysis methods for estimating the effect of these policies on firearm deaths, which may help in the evaluation of whether estimates from prior research can be considered to be accurate. The results suggest substantial statistical problems with many of the methods used. The authors also identify the best method among those assessed.

This report should be of interest to researchers familiar with statistical methods for estimating causal effects in longitudinal time series data, those who are trying to understand the effects of gun policies as revealed in the existing literature, or those who are planning new studies that use statistical models to investigate these effects.

Key Takeaways

  • Simulation results reveal that many commonly used modeling approaches in gun policy research have quite poor type 1 error rates.
  • Several models have type 1 error rates ten times greater than the nominal α = 0.05 that was intended.
  • Huber and cluster adjustments often do not fix these problems, and Huber adjustments can sometimes make them worse.
  • The models also had surprisingly low statistical power to detect an effect-sized equivalent to a change of 1,000 deaths per year if a law were implemented nationally.
  • Most models could correctly reject the null hypothesis only about 10 percent of the time with this true effect. With power this low, a large fraction of effects that are statistically significant will be found to be in the opposite direction as the true effect, and all significant effects will greatly exaggerate the magnitude of the true effect.
  • One model was identified as having the best performance across all assessed criteria. This model is a negative binomial model of firearm deaths that includes time-fixed effects, an autoregressive effect, and change coding for the law effect. The preferred specification includes no state-fixed effects or standard error adjustment.

Recommendations

  • Researchers should consider using Bayesian statistical methods when estimating the effect of state laws on firearm death rates.
  • Given the lack of power to conduct traditional significant testing, policymakers will be well served to understand the range of possible effects associated with a given policy and where the weight of current evidence lies.
  • To correctly estimate the uncertainty in the model estimates, the models may need to include an autoregressive term. However, this requires careful consideration of how effects are coded to avoid dramatically biased effect estimates.

Topics

Document Details

Citation

Chicago Manual of Style

Schell, Terry L., Beth Ann Griffin, and Andrew R. Morral, Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study. Santa Monica, CA: RAND Corporation, 2018. https://www.rand.org/pubs/research_reports/RR2685.html.
BibTeX RIS

This project is a RAND Venture. Funding was provided by gifts from RAND supporters and income from operations.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.