Researchers can use sensitivity analyses to evaluate whether they, or stakeholders, should be careful in extending their findings on school choice programs to populations or locations beyond those studied.
Can Broad Inferences Be Drawn from Lottery Analyses of School Choice Programs?
An Exploration of Appropriate Sensitivity Analyses
Published in: Journal of School Choice: International Research and Reform, v. 10, no. 1, Mar. 2016, p. 48-72
Posted on RAND.org on April 01, 2016
Research Question
- How can researchers test their analyses of school choice programs for indications that they, or media and policy stakeholders, should be careful about making broad inferences from the results?
School choice programs continue to be controversial, spurring a number of researchers into evaluating them. When possible, researchers evaluate the effect of attending a school of choice using randomized designs to eliminate possible selection bias. Randomized designs are often thought of as the gold standard for research, but many circumstances can limit external validity of inferences from these designs in the context of school choice programs. In this article, we examine whether these limitations are applicable to previous evaluations of voucher, charter schools, magnet, and open-enrollment programs. We devise simple sensitivity analyses that researchers could conduct when analyzing lotteried programs to determine whether there are reasons to be cautious about the breadth of appropriate inferences.
Key Findings
- Studies of school choice programs that use a lottery-based admission process frequently are susceptible to misinterpretation.
- Some journalists, policymakers, and other stakeholders may not understand the limitations of a given study and erroneously state that its findings apply to other types of students or schools.
- Researchers should evaluate their studies for issues with external validity and more clearly state whether caution is needed in making inferences from the results.
Recommendation
Researchers should evaluate whether their findings are externally valid by using the sensitivity analysis tools discussed in the article.
This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.