The authors use simulations to assess the performance of a wide range of statistical models commonly used in the gun policy literature to estimate the effects of state-level gun policies on firearm deaths and to identify the most-appropriate statistical methods for producing estimates. The results suggest substantial statistical problems with many of the methods used in this field. The authors identify the best method among those assessed.

Cover: Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths
Read Online

Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths

A Simulation Study

by Terry L. Schell, Beth Ann Griffin, Andrew R. Morral

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 1.2 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Download Support Files

Data Files

Simulation code and data in R formats

FormatFile SizeNotes
zip file 0.4 MB

The file(s) provided above are ZIP-formatted archives, which most modern systems can natively unpack. If your computer does not unpack the archive when you double-click it, you may need to use a separate decompression program such as UnZip.

Purchase

Purchase Print Copy

 FormatList Price Price
Add to Cart Paperback112 pages $19.00 $15.20 20% Web Discount

Research Question

  1. What statistical methods are most appropriate to analyze the effects of gun policies?

The RAND Corporation launched its Gun Policy in America initiative with the goal of creating objective, factual resources for policymakers and the public on the effects of gun laws. As a part of this project, RAND researchers conducted a systematic literature review and evaluation of scientific studies on the effects of 13 classes of policies. One of the findings of the review was that the effects of policies estimated in the literature appeared to be sensitive to the specific statistical methods that were employed. This suggests the importance of identifying the most-appropriate statistical methods to use on these data.

In this report, the authors use simulations to assess the performance of a wide range of statistical models commonly used in the gun policy literature to estimate the effects of state-level gun policies on firearm deaths. The study aimed to identify the most-appropriate statistical modeling and analysis methods for estimating the effect of these policies on firearm deaths, which may help in the evaluation of whether estimates from prior research can be considered to be accurate. The results suggest substantial statistical problems with many of the methods used. The authors also identify the best method among those assessed.

This report should be of interest to researchers familiar with statistical methods for estimating causal effects in longitudinal time series data, those who are trying to understand the effects of gun policies as revealed in the existing literature, or those who are planning new studies that use statistical models to investigate these effects.

Key Findings

  • Simulation results reveal that many commonly used modeling approaches in gun policy research have quite poor type 1 error rates.
  • Several models have type 1 error rates ten times greater than the nominal α = 0.05 that was intended.
  • Huber and cluster adjustments often do not fix these problems, and Huber adjustments can sometimes make them worse.
  • The models also had surprisingly low statistical power to detect an effect-sized equivalent to a change of 1,000 deaths per year if a law were implemented nationally.
  • Most models could correctly reject the null hypothesis only about 10 percent of the time with this true effect. With power this low, a large fraction of effects that are statistically significant will be found to be in the opposite direction as the true effect, and all significant effects will greatly exaggerate the magnitude of the true effect.
  • One model was identified as having the best performance across all assessed criteria. This model is a negative binomial model of firearm deaths that includes time-fixed effects, an autoregressive effect, and change coding for the law effect. The preferred specification includes no state-fixed effects or standard error adjustment.

Recommendations

  • Researchers should consider using Bayesian statistical methods when estimating the effect of state laws on firearm death rates.
  • Given the lack of power to conduct traditional significant testing, policymakers will be well served to understand the range of possible effects associated with a given policy and where the weight of current evidence lies.
  • To correctly estimate the uncertainty in the model estimates, the models may need to include an autoregressive term. However, this requires careful consideration of how effects are coded to avoid dramatically biased effect estimates.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Methods

  • Chapter Three

    Results

  • Chapter Four

    Discussion

  • Appendix A

    Technical Description of Evaluated Models

  • Appendix B

    Standard Error Correction Factors

This project is a RAND Venture. Funding was provided by gifts from RAND supporters and income from operations.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.