Assessing the Impact of Comprehensive School Reform: Have We Put the Cart Before the Horse?

With pressure mounting over the last 20 years to improve how well students do in school, many schools have turned to comprehensive school reform, or CSR. CSR is one in the limited set of interventions that the No Child Left Behind Act explicitly allows. CSR is based on the idea that a school should have a coherent vision of its mission and educational strategy that aligns every aspect of its operations—including curriculum, methods of teaching, forms of governance, and parental involvement—into a well-functioning delivery system. So far, more than 8,000 schools have adopted CSR models and more than $2 billion in federal funds have been used to implement various CSR strategies.

Not surprisingly, the debate over CSR has focused on how effective it has been in improving student performance. Research results to date have been mixed, with some studies showing that CSR has had a modest effect and others showing that it has had no effect. But are we asking the right question? One of the keys to determining the effectiveness of any intervention is whether the intervention has been delivered as its designers intended. However, a major shortcoming of all the research on CSR effectiveness is that the research does not first address the question of whether schools have actually implemented their chosen CSR model. Without knowing the level of implementation, researchers cannot determine effectiveness.

A new RAND Corporation study seeks to answer this question, developing a unique approach to quantitatively measure the level of CSR implementation and then using it to measure the implementation of four different CSR models in a large number of schools. The approach uses a survey of principals and teachers in a sample of 250 model schools in two states (Florida and Texas), supplemented with in-depth case studies of 12 schools—an approach that allows for a much larger sample of CSR schools to be included in the study and that also allows researchers to include non-CSR schools for comparison purposes.

The study finds that few CSR schools have fully implemented all the core components of their CSR models as envisioned by the designers. Some core components, such as prescribed curriculum, were more fully implemented than others, such as prescribed instructional practices or practices to increase parental involvement.

The study also finds that CSR implementation support falls short of the levels recommended by the model designers. Such support falls into two categories: external (e.g., providing for principal and teacher consultation with model designers and teacher training) and internal (e.g., appointing a school staff member to facilitate and coordinate implementation). In general, teachers reported only a lukewarm commitment to implementing the school's model, with most teachers feeling that the training they received did not fully prepare them to start using the model.

One of the more surprising findings was that, on average, all schools in the survey engaged in many of the same activities at the same frequency and level of intensity, regardless of whether those schools used one of the four CSR models. Only a handful of practices differed between types of model schools and between model schools and their matched non-model schools.

The findings, if they are replicated in future studies, have several broad implications. The most important is that, at the current level of implementation, CSR models are likely to have modest or no effects on student achievement.

However, there is significant room to increase the level of implementation of CSR models. For instance, schools could ensure that teachers are committed to implementing the adopted model and that they receive the necessary initial training. Also, teachers could be more involved in choosing the model, and professional development could be increased initially and maintained for as long as the school uses the model. Without such support, it is clear that the implementation of CSR models cannot fully enable schools meet the No Child Left Behind goal of 100-percent proficiency in reading and mathematics by 2014.

Finally, the findings underscore the importance of accounting for the level of implementation when seeking to measure CSR effects on student achievement. The measurement approach used in this study can be used in future studies to measure a broader sample of schools. Researchers cannot determine whether a CSR model affects student achievement until they first know whether and how completely the model has been implemented.

READ THE RESEARCH BRIEF: Comprehensive School Reform: The Implementation Gap

The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world.

RAND Education conducts research on a wide range of topics in education. Our mission is to bring accurate data and careful, objective analysis to the national debate on education policy. To access all of our research, please visit our web site at www.rand.org/education/.

Learn More >

Copyright © 2006 RAND Corporation. RAND® is a registered trademark.

The RAND Education Congressional Newsletter is distributed on an opt-in basis only by visiting us at www.rand.org. We respect your privacy. If you do not wish to receive this periodic newsletter, please email ocr@rand.org or call (703) 413-1100 x5320.

This newsletter is also available on the RAND Congressional web site »