Can Broad Inferences Be Drawn from Lottery Analyses of School Choice Programs?
An Exploration of Appropriate Sensitivity Analyses
Published in: Journal of School Choice: International Research and Reform, v. 10, no. 1, Mar. 2016, p. 48-72
Posted on RAND.org on April 01, 2016
Read MoreAccess further information on this document at Journal of School Choice: International Research and Reform
This article was published outside of RAND. The full text of the article can be found at the link above.
- How can researchers test their analyses of school choice programs for indications that they, or media and policy stakeholders, should be careful about making broad inferences from the results?
School choice programs continue to be controversial, spurring a number of researchers into evaluating them. When possible, researchers evaluate the effect of attending a school of choice using randomized designs to eliminate possible selection bias. Randomized designs are often thought of as the gold standard for research, but many circumstances can limit external validity of inferences from these designs in the context of school choice programs. In this article, we examine whether these limitations are applicable to previous evaluations of voucher, charter schools, magnet, and open-enrollment programs. We devise simple sensitivity analyses that researchers could conduct when analyzing lotteried programs to determine whether there are reasons to be cautious about the breadth of appropriate inferences.
- Studies of school choice programs that use a lottery-based admission process frequently are susceptible to misinterpretation.
- Some journalists, policymakers, and other stakeholders may not understand the limitations of a given study and erroneously state that its findings apply to other types of students or schools.
- Researchers should evaluate their studies for issues with external validity and more clearly state whether caution is needed in making inferences from the results.
Researchers should evaluate whether their findings are externally valid by using the sensitivity analysis tools discussed in the article.