Education evaluations that analyze student achievement typically use baseline scores to control for unobserved ability. These analyses rely on the assumption that the conditions under which the test was administered are consistent from pretest to posttest. We caution evaluators to examine this assumption, provide a framework for evaluation under changes in test conditions, and demonstrate an application of this framework. We examine a case in which treatment status is correlated with the changes in test durations. We test three estimation strategies to account for this, each finding bias in estimation of treatment effects when not accounting for the changes in test conditions. We also discuss what to do in absence of such information.
Baird, Matthew D. and John F. Pane, Controlling for Changes in Test Conditions when Estimating Education Intervention Effects. Santa Monica, CA: RAND Corporation, 2018. https://www.rand.org/pubs/working_papers/WR1245.html.
Baird, Matthew D. and John F. Pane, Controlling for Changes in Test Conditions when Estimating Education Intervention Effects, RAND Corporation, WR-1245-BMGF, 2018. As of January 31, 2023: https://www.rand.org/pubs/working_papers/WR1245.html