A number of challenges are encountered when evaluating a large-scale, multisite educational reform aimed at changing classroom practice. The challenges include substantial variability in implementation with little information on actual practice; lack of common, appropriate outcome measures; and the need to synthesize evaluation results across multiples study sites. This article describes an approach to addressing these challenges in the context of a study of the relationships between student achievement and instructional practices in the National Science Foundation’s Systematic Initiatives (SI) program. The authors gathered data from 11 SI sites and investigated relationships both at the site level and pooled across sites, using a planned meta-analytic approach. They found small but consistent positive relationships between teachers’ reported use of standards-based instruction and student achievement. The article also describes how the authors addressed the challenges discussed, as well as a number of additional obstacles that need to be addressed to improve future evaluations of large-scale reforms.
Originally published in: Educational Evaluation and Policy Analysis, v. 25, no. 1, Spring 2003, pp. 1-29.
This report is part of the RAND Corporation Reprint series. The Reprint was a product of the RAND Corporation from 1992 to 2011 that represented previously published journal articles, book chapters, and reports with the permission of the publisher. RAND reprints were formally reviewed in accordance with the publisher's editorial policy and compliant with RAND's rigorous quality assurance standards for quality and objectivity. For select current RAND journal articles, see External Publications.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.