Cover: Using State School Accountability Data to Evaluate Federal Programs:  A Long Uphill Road

Using State School Accountability Data to Evaluate Federal Programs: A Long Uphill Road

Published 2003

by Sheila Nataraj Kirby, Daniel F. McCaffrey, J. R. Lockwood, Jennifer Sloan McCombs, Scott Naftel, Heather Barney

Purchase Print Copy

Add to Cart Paperback24 pages Free

Evaluations of federal programs designed to improve student achievement generally depend on data gathered by the states for school accountability purposes, rather than data specifically designed for program evaluation. In addition, these data are available at the school level but not at the student level. This article first discusses issues related to the quality of school-level data collected as part of state accountability systems, including the reliability and validity of school-level test scores as a measure of the value added by schools to student learning. It then outlines various ways in which school-level data can be usefully analyzed and illustrates the challenges inherent in doing so, including the challenges of aggregating data across states to find an overall program effect. The final section discusses the implications of the arguments presented here for measuring changes in school performance and linking these effects to a specific program. Ultimately, our ability to measure changes in outcomes and link them back to the intervention depends on three factors: (a) identifying a set of activities attributable to the program, (b) measuring the quality of implementation of these activities, and (c) obtaining a valid and reliable measure of the desired outcome. The article makes it clear that none of these is easy to come by.

Originally published in: Peabody Journal of Education, v. 77, no. 4, 2002, pp. 122-145.

This report is part of the RAND reprint series. The Reprint was a product of RAND from 1992 to 2011 that represented previously published journal articles, book chapters, and reports with the permission of the publisher. RAND reprints were formally reviewed in accordance with the publisher's editorial policy and compliant with RAND's rigorous quality assurance standards for quality and objectivity. For select current RAND journal articles, see External Publications.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.