A Decade of Whole-School Reform
NAS's core premise was that all high-quality schools possess, de facto, a unifying design that enables all staff members to function to the best of their abilities and that integrates research-based practices into a coherent and mutually reinforcing set of effective approaches to teaching and learning. The best way to ensure that lower-performing schools adopted successful designs was to fund teams to develop "break the mold" school designs that could readily be adopted by communities nationwide. After developing the designs, the teams would then implement them in schools throughout the country. This adoption process would fulfill NAS's primary goal of improving the performance of large numbers of students.
This whole-school design approach to better educational outcomes was a dramatically different way of initiating and disseminating large-scale educational improvements to many schools across the country. It was a unique combination of (1) private sector involvement using a venture capitalist approach; (2) the choice of whole-school design as a vehicle for reform; and (3) the ambitious goal of achieving scale-up (disseminating designs to a large number of schools in a jurisdiction that agreed to partner with NAS).
The NAS experimental approach required careful development and demonstration of designs before moving to the scale-up stage; therefore, NAS proceeded in phases:
- Competition and selection phases (1992)
- One-year development phase (1992-1993)
- Two-year demonstration phase (1993-1995, including the 1993-1994 and 1994-1995 school years)
- Three-year scale-up phase (1995-1998, from the school year starting in 1995 to the school year ending in 1998).
RAND'S ROLE IN THE NAS INITIATIVENAS asked RAND to provide analytic support to its school reform efforts. This support took many forms but primarily was intended to document and analyze the conditions under which NAS progressed toward its goals of widespread implementation of its designs and improved student performance associated with the use of those designs. The support from RAND included the following tasks:
- Documenting NAS's efforts to assess its contributions to education reform
- Describing the designs and analyzing changes in them over time
- Assessing the level of implementation in design-based schools during the demonstration and scale-up phases
- Identifying factors that impede or encourage implementation in the demonstration and scale-up phases
- Measuring whether the adoption of school designs resulted in the outcomes desired by NAS and its partnering districts in the scale-up phase.
- A longitudinal sample of more than 100 NAS schools that began implementation early on in the scale-up phase, including data on implementation and performance from principals, teachers, and districts
- Case studies in 40 schools to analyze implementation and the role that districts play in impeding or enabling comprehensive school reform
- A description of how designs have evolved from the
initial proposal stage to implementation at scale-up in schools across the nation
- Analyses in one urban school district of how whole-school designs promote changes in classroom instruction, teaching and learning, and individual-level student achievement scores
- An analysis of performance differences among high-implementing NAS sites and ongoing discussions with NAS staff and design team leaders.
- Authentic Teaching, Learning, and Assessment for All Students (ATLAS)
- Audrey Cohen College (AC) (currently renamed Purpose-Centered Education)
- Co-NECT Schools (CON)
- Expeditionary Learning Outward Bound (EL)
- Modern Red Schoolhouse (MRSH)
- National Alliance for Restructuring Education
(NARE) (currently renamed America's Choice Design Network)
- Roots & Wings (RW).
- The design itself and its ability to offer coherent, comprehensive, and consistent education programs, and the assistance offered to schools by the design teams to ensure implementation
- The efficacy of the selection and matching of designs
and schools to ensure that teachers "buy in" to the
- The capacity of the specific schools for undertaking the reform, including the schools' other efforts at reform, educational leadership, and teaching capability
- School-specific demographics, enrollment and grade level of the schools, and climate
- The context set by the district, including the existing infrastructure support and incentives for design implementation and improved student performance
- Other factors, such as the context set by state policies for testing and accountability, community influences, and NAS funding policies.
- For the entire sample of schools RAND surveyed, implementation in the scale-up schools increased modestly from 1997 to 1999. The between-school variance in the level of implementation decreased somewhat over time, but the within-school variance increased. There was much greater variance in implementation within schools than between schools, suggesting that designs had not become "schoolwide."
- There were large differences in implementation by jurisdiction, by design, and across schools.
- For schools newly adopting designs, implementation increased and expanded over the first four years after the schools adopted designs, although at decreasing rates each year.
- Among the four jurisdictions with ten or more of its schools implementing NAS, Memphis and Kentucky appeared to be the most successful in terms of
improvement in mathematics test scores, whereas Cincinnati and Washington schools did better in read-ing scores.
- In total, of the 163 schools for which we had data allowing comparisons in performance relative to the district or state, 81 schools (50 percent) made gains in mathematics relative to the district and 76 schools (47 percent) made gains in reading relative to the district.
- Strong leadership by a principal as reported by teachers had significant positive effects on students' state test scores in reading and mathematics.
- Instructional conditions promoted by reforms such as NAS--including teacher-reported collaboration, qual-
ity of professional development, and reform-like
instructional practices--were not related to student achievement, net of other student and classroom con-
- In general, early implementation of NAS designs in a high-poverty district that exists within a high-stakes accountability system did not result in significant effects on student achievement.
- NAS funding and leadership led to the deliberate development of several functioning whole-school design teams.
- NAS showed that external change agents that depended initially on external funding could be moved toward self-sufficiency over time.
- NAS explicitly sought scale-up of the reform initiative.
- NAS actions as a change agent have significantly influenced policy in its areas of interest.
- NAS explicitly made analysis of design implementation and good consumer education a part of its efforts.
- The initial hypothesis--that a school could improve its performance by adopting a whole-school design--was largely unsupported. Our general findings showed difficulties
in design implementation and a lack of significant improvement in school performance in a large percentage of the schools in our samples.
- The hypothesis that the designs alone were not helpful to schools and that schools needed assistance in implementation was proven correct. Teachers and school administrators clearly reported higher levels of implementation associated with strong assistance from design teams. Just as important, and consistent with the implementation literature, conditions within the schools and districts and the manner of selection also proved to be important to implementation and outcomes.
- The scale-up hypothesis--that a district that converted 30 percent of its schools using whole-school approaches would become high performing and not revert to unproductive practices--was disproved. Districts such as Memphis quickly reverted to their former status after changes in their administrations.
- The scale-up hypothesis--that a district needs to provide a supportive environment--was dramatically proven. Barriers to implementation reported by school staff focused on unsupportive district practices. They also pointed to the challenges inherent in simultaneously adopting multiple reforms, high-stakes testing regimes, and new school designs.
- In general, we conclude that NAS's initial theory of action was largely underdeveloped and nonspecific in terms of ensuring a successful scale-up experience. The causal chain of events leading to a strong implementation and positive outcomes has proven to be far more complex than what was originally envisioned by NAS and one that remains largely outside of NAS's control and influence. This finding is in keeping with the literature on implementation indicating the complexity of the change process.
- Externally developed education reform interventions cannot "break the mold" and still be marketable and be able to implemented in current district and school contexts. Schools are not, by and large, fertile ground for "break the mold" ideas, often because of a lack of capacity or because of local, state, or district regulations.
- External interventions need to address systemic issues that can hinder implementation, such as lack of teacher capa-city to undertake implementation of the designs, lack
of leadership from the school principal, or lack of a coherent district infrastructure.
- A rush to scale-up when interventions are not completely developed weakens the results. This problem is likely to persist partly because developers are under financial pressure to scale up their interventions before they are thoroughly evaluated and partly because districts and schools are under extreme political pressure to adopt external solutions, proven or not, as a means of addressing the lackluster performance of their students.
- Key components of successful implementation are consistent, clear, and frequent communication and assistance between design developers and the schools, particularly the schools' teachers.
- Monitoring site progress, self-evaluation, and assessment of outcomes are necessary if external developers are to be successful and improve their offerings over time.
- The typical outcome measures used in public accountability systems provide very limited indicators of student and school performance. The use of state- and district-mandated assessments is likely to present a constant hindrance to understanding the effect of innovative approaches unless alternative indicators and assessments are developed in ways that align well with the reforms' goals.
The high-stakes testing regimes currently in vogue and the overwhelming emphasis given to scores on state- and district-mandated tests as the measures of student improvement do not bode well for many innovative reform efforts.
A BRIEF HISTORY OF NAS FOCUSING ON ITS THEORY OF ACTIONAny educational reform must have two components: a theory of learning (that brings together assumptions about how students learn, instructional strategies, and performance) and a theory of action (that focuses on the conditions under which the reform will work). The theory of learning in the NAS initiative was embodied in the individual designs.
Initially, NAS's theory of action was quite simple and undeveloped: Designs would be developed, schools would adopt them in some unspecified manner, and this adoption would improve student outcomes. NAS helped to create the designs through a request for proposals process from October 1991 to July 1992 during which more than 600 teams applied. NAS chose 11 design teams for initial development. From July 1992 to July 1993, those teams worked to further develop their theories of learning. A year later, NAS reduced the number of teams from 11 to 9 in part due to funding difficulties. The two teams that NAS removed had a district as their team leader. This decision indicated that NAS's theory of action did not include locally based and led teams, in part because of its ambition for national scale-up.
The demonstration phase, from 1993 to 1995, took place in 147 schools that the design teams selected to partner with them. RAND's analysis of the progress at that point indicated that an important component of the design team intervention was the assistance that teams provided to schools, which enabled those schools to adopt the designs. NAS calls this component of the initiative "design-based assistance." NAS's theory of change evolved to include the notion that design teams would provide not just designs but also design-based assistance to implement those designs successfully.
RAND's analysis of the progress in the demonstration schools also showed that school and district factors had a strong relationship to the success of an implementation. NAS by then understood that to succeed it would have to ensure a supportive environment for its designs. During the demonstration phase, NAS removed two more teams, both of which did not have national scale-up intentions and which appeared at the time to have characteristics that were specific to districts or states.
NAS outlined its scale-up strategy--a third iteration of its theory of action. Because of its experiences during the demonstration phase, NAS became more aware of the importance of gaining district-level support for teams and providing a supportive district structure, including resources, school autonomy, and professional development for the schools. Thus, it chose a "concentration" strategy--transforming a few districts by having a significant number of schools within those districts adopt design-based assistance.
Ten jurisdictions agreed to partner with NAS. Those jurisdictions promised to get approximately 30 percent of their schools to use the designs within a three-year period and to provide a supportive environment for the schools' efforts. The 30-percent figure was somewhat arbitrary. The theory of action was that if NAS could implement its designs in about a third of a district's schools, the district itself would have to change its policies to become permanently more supportive of its schools. In addition, NAS insisted that the design teams should become more self-sufficient at this point. Design teams would charge the districts fees for the design-based assistance, and NAS would work with districts to help make a supportive environment evolve. NAS promised districts that that by using its designs, district schools would be able to show dramatic improvements on standardized district-mandated tests within the three-year period.
The seven teams entering the scale-up phase included the following:
CONCEPTUAL FRAMEWORK OF THE STUDIESThe conceptual framework underpinning the RAND studies of NAS draws from previous research on implementation of school reforms and educational change. A critical assumption underlying the designs is that coherent, focused, and sustained implementation of key design components (including professional development, curriculum and instructional materials, content and performance standards, assessments, organization and governance, and parent and community involvement) will eventually change school and classroom learning environments and thereby students' academic outcomes. Implementation consists of the process of putting into practice the elements or sets of activities defined by design teams as core components of their designs.
However, throughout the history of educational reform efforts, a consistent theme has been that the process of planned educational change is much more complex than what was initially anticipated. Past research on external change agents as reform mechanisms in K-12 education has shown that as these externally developed interventions are implemented, they tend to change significantly as they adapt to local conditions and contexts or progress through scaling up. Often, those conducting the intervention develop strategies to assist schools in understanding and implementing the intervention. Despite these measures, implementation tends to vary across sites, and the outcomes--in terms of the desired change--also vary considerably. This variation in outcomes occurs largely because of the number of players involved and the number of factors that need to be in alignment to support such fundamental change.
The factors that could be expected to affect both implementation and outcomes include the following, some of which were not readily controlled by NAS or its design teams:
FINDINGSHere we summarize the specific findings of the various analyses. The findings are grouped into three broad areas: (1) the evolution of designs, (2) the level of implementation observed in the NAS sites and the factors related to implementation, and (3) student outcomes.
Overview of Changes to Designs over TimeThese findings are based on a continuous tracking of the designs and design teams throughout the analytic effort (Bodilly, 2001). Consistent with the literature on external change agents, the designs adapted to local conditions over time. Some of the design developments were beneficial for enabling schools to improve. For example, the growth in assistance packages, the further development of curricular units, and the development of protocols for a school's choice of design all appeared to be positive adaptations of the designs. Development of basic skills curriculum could also be considered a positive adaptation when well integrated with the principles of the design and not simply a quick add-on to meet district demands.
Other changes to the designs, while understandable, became problematic. Adaptation to district and school policies led to some designs accepting an unaligned and incoherent mix of standards, assessments, curriculum, instruction, and professional development. This situation also allowed for considerable local adaptation to the point that one might question the unifying or coherent nature of a design.
Because changes were occurring to some designs over the demonstration and scale-up phases, a major component of NAS's theory of action--a coherent and unifying design--was often missing or was constantly undergoing revision. It cannot be emphasized enough that during the entire period of the RAND studies, the designs were still in a state of development and evolution. Consistent with the literature, implementation assistance (that is, design-based assistance) became an important part of the intervention as time went on.
Overview of the Implementation Findings
RAND studied the implementation of designs in both the demonstration and scale-up phases. The purpose of the studies was to measure the level of implementation in schools and to determine the conditions under which implementation prospered (Bodilly, 1998; Keltner, 1998; Bodilly and Berends, 1999; Berends, 2000; Berends et al., 2001; Kirby et al., 2001; and Berends, Bodilly, and Kirby, 2002).
The demonstration schools showed some promise, but RAND identified many barriers to implementation (Bodilly et al., 1995; Mitchell, 1996). During scale-up, NAS and the design teams partnered with schools and districts that were characterized by a host of problems related to poverty, achievement, and school and district characteristics (Berends, 1999). Achieving high levels of implementation within those schools across all teachers proved to be challenging. The case study analyses found that by two years into implementation approximately half of the sample sites were implementing designs at a level consistent with the expectations of NAS and the design teams. The other half were implementing at a rate below that level. All sites reported many barriers to further implementation.
The longitudinal sample of teachers supported the findings of the case studies that showed weak implementation and stagnation:
Designs and Design-Based Assistance. Implementa-tion varied by design. In the longitudinal survey sample, schools with CON, NARE, and RW designs clearly reported higher implementation levels. The high levels of implementation reported in all the studies were related to clear communication with and strong assistance from the design teams. In the last of the case study analyses, which used a small sample of schools, we found that several design teams did not have a large amount of information about their schools nor had they adequately measured the level of implementation at the schools. Taken together, the evidence shows that several design teams need to make significant improvements to ensure the implementation of their designs.
Selection Process. A case study in Bodilly (1998) revealed that those school administrators and teachers who felt that their choices were well informed reported higher levels of implementation at their schools than those who reported either being forced to accept a design or not understanding the nature of the design. In the longitudinal survey sample, teacher support and buy-in were related to higher levels of implementation and vice versa. The case study work in Bodilly (1998) also revealed that school principals played an important role in ensuring both a sound selection process and teacher buy-in.
School Capacity. School capacity was important in supporting implementation. The leadership of a principal was a key contributor to implementation across all the studies. The survey analyses indicated that teachers viewed a principal's leadership as the most important predictor of the implementation level achieved. The teacher survey also indicated that teachers' perceptions of students and their readiness to learn were both significantly related to the level of implementation. The various implementation case studies indicated that many other reforms were taking place in the schools while the NAS reform was under way. In many instances, the numerous reforms overloaded teachers and reduced their capacity to implement the designs.
School Context. The teacher surveys revealed that implementation was higher in high-poverty schools and schools serving large numbers of minority students. However, in schools that served significant numbers of both poor and minority students, implementation levels were significantly lower. Across our implementation studies, we found that implementation was higher in elementary schools than in secondary schools and was higher in smaller schools than in larger ones.
District Context. There were large differences in implementation among jurisdictions across all of our studies. In general, implementation was higher in those districts that (1) were more supportive of the NAS designs and were characterized as having a stable district leadership that backed the reform and made it central to its improvement efforts; (2) were not burdened with crises such as budget shortfalls or redistricting; (3) had a coherent program of reform; (4) had resources dedicated to the implementation effort; (5) had significant school-level autonomy; and (6) had trusting relationships among school, district, and union staff.
Several case study analyses highlighted significant barriers to implementation that were inherent in district and union policies. The San Antonio classroom study gave particular insight into how district-level behaviors affected schools and their willingness to undertake and implement designs (Berends et al., 2002). Teachers' views of the consistency and effectiveness of district leadership proved to be positively associated with the teachers' efforts toward implementation. However, multiple reforms easily overwhelmed teachers in their efforts at adopting a new school design. Most disruptive was the high-stakes state testing regime, which inadvertently encouraged a specific focus on basic skills and which resulted in the district adopting specific curricular programs in addition to implementing the designs. Unfortunately, these curricular programs conflicted with the designs and resulted in lower levels of implementation.
Related RAND research and exit interviews with the principals of schools that had dropped the NAS designs highlighted the importance of adequate funding. Lack of funding was the single most important reason cited by most schools in their decision to drop a design. The costs associated with the reform were far greater than the fees paid to design teams. In fact, those fees made up less than one-third of the schools' implementation costs. Significant costs were also borne by teachers in terms of their time and effort in implementing these reforms.
Overview of Student Outcome FindingsOur analysis of performance trends across the set of schools three years into scale-up focused on whether NAS schools made any gains in test scores relative to their respective jurisdictions. While these school-level measures allowed us to compare performance in NAS schools with that of the district as a whole, they are subject to significant limitations. For example, the aggregated measures may fail to capture changes in the tails of a distribution or may miss some significant achievement effects that would be captured if longitudinal student-level data were available and comparable across jurisdictions. In terms of student outcomes, we found the following:
Because of the wide variation in implementation that occurred and the variety of environments that exist within schools and among jurisdictions, it may have been too early to expect robust performance results across the NAS sites. Better and longer-term performance data are needed in order to make conclusive judgments about the various designs and their effects on school performance.
However, our implementation analysis showed that the overall level of implementation increased modestly over time, and there was continuing within-school variation in implementation. If the NAS approach to school improvement is correct, limited implementation will lead to weak effects on student performance. Our findings suggest that we cannot expect to have stronger performance results unless implementation significantly increases.
The detailed classroom study of the San Antonio jurisdiction allowed us to examine whether variation in instructional conditions was related to student achievement, when controlling for other student, teacher, classroom, and school characteristics. Our findings were similar to those of other analyses of schoolwide reforms:
CONCLUSIONSIn the end, the set of RAND analyses illuminated three important areas. First, the analyses provided evidence that an external agent such as NAS could create and promote design teams. Second, the analyses indicated that some of the theory of action behind the NAS efforts was under-developed, and they pointed to important conditions for implementation and improved student outcomes. Third, the analyses offered important lessons on how to carry out future efforts at reform.
The Contribution of New American Schools to Educational ReformNAS did accomplish several of the goals it had set for itself and in the process made a number of important contributions to educational reform that should be kept in mind. These include:
NAS's Theory of ActionThe RAND findings provide mixed evidence to support NAS's theories of positively changing schools with whole-school designs.
Implications for Future EffortsBased on our experience with NAS, we offer the following observations for any future attempts at scaling up school reforms.
BibliographyBerends, Mark. (1999). Assessing the Progress of New American Schools: A Status Report. Santa Monica, CA: RAND (MR-1085-ED).
Berends, Mark. (2000). "Teacher-Reported Effects of New American Schools' Designs: Exploring Relationships to Teacher Background and School Context," in Educational Evaluation and Policy Analysis, 22(1), pp. 65-82.
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (2002). Facing the Challenges of Whole-School Reform: New American Schools After a Decade. Santa Monica, CA: RAND (MR-1498-EDU).
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (Forthcoming [b]). "District and School Leadership for Whole-School Reform: The Experience of New American Schools." In Leadership for School Reform: Lessons from Comprehensive School Reform Designs, edited by J. Murphy and A. Datnow, Thousand Oaks, CA: Corwin Press.
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (Forthcoming ([a]). "Reforming Whole Schools: Challenges and Complexities." In Bringing Equity Back, edited by J. Petrovich and A. W. Wells. (n.p.).
Berends, Mark, JoAn Chun, Gina Schuyler, Sue Stockly, and R. J. Briggs. (2002). Conflicting School Reforms: Effects of New American Schools in a High-Poverty District. Santa Monica, CA: RAND (MR-1483-EDU).
Berends, Mark, Sheila Nataraj Kirby, Scott Naftel, and Christopher McKelvey. (2001). Implementation and Performance in New American Schools: Three Years into Scale-Up. Santa Monica, CA: RAND (MR-1145-EDU).
Bodilly, Susan J. (1996a). Lessons from New American Schools Development Corporation's Demonstration Phase. Santa Monica, CA: RAND (MR-729-NASDC).
Bodilly, Susan J. (1996b). "Lessons Learned from RAND's Formative Assessment of NASDC's Phase 2 Demonstration Effort." pp. 289-324 in Bold Plans for School Restructuring: The New American Schools Designs, edited by Sam Stringfield, Steven Ross, and Lana Smith. Mahwah, NJ: Lawrence Erlbaum Associates.
Bodilly, Susan J. (1998). Lessons from New American Schools' Scale-Up Phase: Prospects for Bringing Designs to Multiple Schools. Santa Monica, CA: RAND (MR-1777-NAS).
Bodilly, Susan J. (2001). New American Schools' Concept of Break the Mold Designs: How Designs Evolved over Time and Why. Santa Monica, CA: RAND (MR-1288-NAS).
Bodilly, Susan J., and Mark Berends. (1999). "Necessary District Support for Comprehensive School Reform." pp. 111-119 in Hard Work for Good Schools: Facts Not Fads in Title I Reform, edited by Gary Orfield and Elizabeth H. DeBray. Boston, MA: Civil Rights Project, Harvard University.
Bodilly, Susan J. and Thomas K. Glennan. (1998). Reforming America's Schools: Observations on Implementing "Whole School Designs." Santa Monica, CA: RAND (RB-8016-EDU).
Bodilly, Susan J., Susanna Purnell, Kimberly Ramsey, and Christina Smith. (1995). Designing New American Schools: Baseline Observations on Nine Design Teams. Santa Monica, CA: RAND (MR-598-NASDC).
Chun, JoAn, Brian Gill, and Jodi Heilbrunn. (2001). The Relationship Between Implementation and Achievement: Case Studies of New American Schools. Santa Monica, CA: RAND (DRU-2562-EDU).
Glennan, Thomas K. Jr. (1998). New American Schools After Six Years. Santa Monica, CA: RAND (MR-945-NASDC).
Keltner, Brent R. (1998). Funding Comprehensive School Reform. Santa Monica, CA: RAND (IP-175-EDU).
Kirby, Sheila Nataraj, Mark Berends, and Scott Naftel. (2001). Implementation in a Longitudinal Sample of New American Schools: Four Years into Scale-Up. Santa Monica, CA: RAND (MR-1413-EDU).
Mitchell, Karen. (1996). Reforming and Conforming: NASDC Principals Discuss School Accountability Systems. Santa Monica, CA: RAND (MR-716-NASDC).
RAND research briefs summarize research that has been more fully documented elsewhere. This research brief draws primarily on the latest in the series of reports on the New American Schools. This brief describes work done within RAND Education and documented in Facing the Challenges of Whole-School Reform: New American Schools After a Decade, by Mark Berends, Susan Bodilly, and Sheila Nataraj Kirby, MR-1498-EDU, 2002, 266 pp., ISBN: 0-8330-3133-3.
Building on more than 25 years of research and evaluation work, RAND Education has as its mission the improvement of educational policy and practice in formal and informal settings from early childhood on. A profile of RAND Education, abstracts of its publications, and ordering information may be viewed at www.rand.org.
Abstracts of all RAND documents may be viewed on the web (http://www.rand.org). Publications are distributed to the trade by NBN. RAND® is a registered trademark. RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis; its publications do not necessarily reflect the opinions or policies of its research sponsors.
RAND® is a registered trademark. RAND is a nonprofit institution that helps improve public policy and decisionmaking through research and analysis; its publications do not necessarily reflect the opinions or policies of its research sponsors.
Copyright © 2002 RAND
All rights reserved. Permission is given to duplicate this on-line document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes.
Published 2002 by RAND
RAND Home Page