Jan 1, 2002
Spurred by the piecemeal approach to school reform that had produced little change in student performance, New American Schools (NAS), a private nonprofit organization, launched its efforts for whole-school reform in 1991. NAS's mission was to help schools and districts significantly raise the achievement of large numbers of students through the use of whole-school designs.
NAS's core premise was that all high-quality schools possess, de facto, a unifying design that enables all staff members to function to the best of their abilities and that integrates research-based practices into a coherent and mutually reinforcing set of effective approaches to teaching and learning. The best way to ensure that lower-performing schools adopted successful designs was to fund teams to develop "break the mold" school designs that could readily be adopted by communities nationwide. After developing the designs, the teams would then implement them in schools throughout the country. This adoption process would fulfill NAS's primary goal of improving the performance of large numbers of students.
This whole-school design approach to better educational outcomes was a dramatically different way of initiating and disseminating large-scale educational improvements to many schools across the country. It was a unique combination of (1) private sector involvement using a venture capitalist approach; (2) the choice of whole-school design as a vehicle for reform; and (3) the ambitious goal of achieving scale-up (disseminating designs to a large number of schools in a jurisdiction that agreed to partner with NAS).
The NAS experimental approach required careful development and demonstration of designs before moving to the scale-up stage; therefore, NAS proceeded in phases:
NAS asked RAND to provide analytic support to its school reform efforts. This support took many forms but primarily was intended to document and analyze the conditions under which NAS progressed toward its goals of widespread implementation of its designs and improved student performance associated with the use of those designs. The support from RAND included the following tasks:
These tasks were conducted over the first seven years of the NAS initiative, from the demonstration phase through the scale-up phase, and the results are documented in a series of RAND reports. Over this period, RAND's program of studies has included the following:
Each piece of the RAND program of studies supplied an important building block toward a full understanding of the NAS initiative. Together, these studies provide a cogent, consistent, and comprehensive examination of the NAS initiative and offer useful and timely information to decisionmakers considering or already engaging in whole-school reform. The final report in the series (Berends, Bodilly, and Kirby, 2002) takes a retrospective look at the NAS initiative, drawing together the findings from RAND's many studies and reflecting on the lessons learned from this initiative and the implications for future school reform efforts. This latest report should interest policymakers and practitioners in K–12 education because it offers some important lessons on attempts to improve student performance through external interventions—the focus of the current federally funded Comprehensive School Reform Demonstration program.
Any educational reform must have two components: a theory of learning (that brings together assumptions about how students learn, instructional strategies, and performance) and a theory of action (that focuses on the conditions under which the reform will work). The theory of learning in the NAS initiative was embodied in the individual designs.
Initially, NAS's theory of action was quite simple and undeveloped: Designs would be developed, schools would adopt them in some unspecified manner, and this adoption would improve student outcomes. NAS helped to create the designs through a request for proposals process from October 1991 to July 1992 during which more than 600 teams applied. NAS chose 11 design teams for initial development. From July 1992 to July 1993, those teams worked to further develop their theories of learning. A year later, NAS reduced the number of teams from 11 to 9 in part due to funding difficulties. The two teams that NAS removed had a district as their team leader. This decision indicated that NAS's theory of action did not include locally based and led teams, in part because of its ambition for national scale-up.
The demonstration phase, from 1993 to 1995, took place in 147 schools that the design teams selected to partner with them. RAND's analysis of the progress at that point indicated that an important component of the design team intervention was the assistance that teams provided to schools, which enabled those schools to adopt the designs. NAS calls this component of the initiative "design-based assistance." NAS's theory of change evolved to include the notion that design teams would provide not just designs but also design-based assistance to implement those designs successfully.
RAND's analysis of the progress in the demonstration schools also showed that school and district factors had a strong relationship to the success of an implementation. NAS by then understood that to succeed it would have to ensure a supportive environment for its designs. During the demonstration phase, NAS removed two more teams, both of which did not have national scale-up intentions and which appeared at the time to have characteristics that were specific to districts or states.
NAS outlined its scale-up strategy—a third iteration of its theory of action. Because of its experiences during the demonstration phase, NAS became more aware of the importance of gaining district-level support for teams and providing a supportive district structure, including resources, school autonomy, and professional development for the schools. Thus, it chose a "concentration" strategy—transforming a few districts by having a significant number of schools within those districts adopt design-based assistance.
Ten jurisdictions agreed to partner with NAS. Those jurisdictions promised to get approximately 30 percent of their schools to use the designs within a three-year period and to provide a supportive environment for the schools' efforts. The 30-percent figure was somewhat arbitrary. The theory of action was that if NAS could implement its designs in about a third of a district's schools, the district itself would have to change its policies to become permanently more supportive of its schools. In addition, NAS insisted that the design teams should become more self-sufficient at this point. Design teams would charge the districts fees for the design-based assistance, and NAS would work with districts to help make a supportive environment evolve. NAS promised districts that that by using its designs, district schools would be able to show dramatic improvements on standardized district-mandated tests within the three-year period.
The seven teams entering the scale-up phase included the following:
NAS partnered with the following jurisdictions: Cincinnati; Dade County, Florida; Kentucky; Maryland; Memphis; Philadelphia; Pittsburgh; San Antonio; San Diego; and five districts in the state of Washington. Close to 185 schools partnered with design teams in these jurisdictions. As a whole, NAS design teams spread to more than 550 schools by 1995.
The conceptual framework underpinning the RAND studies of NAS draws from previous research on implementation of school reforms and educational change. A critical assumption underlying the designs is that coherent, focused, and sustained implementation of key design components (including professional development, curriculum and instructional materials, content and performance standards, assessments, organization and governance, and parent and community involvement) will eventually change school and classroom learning environments and thereby students' academic outcomes. Implementation consists of the process of putting into practice the elements or sets of activities defined by design teams as core components of their designs.
However, throughout the history of educational reform efforts, a consistent theme has been that the process of planned educational change is much more complex than what was initially anticipated. Past research on external change agents as reform mechanisms in K–12 education has shown that as these externally developed interventions are implemented, they tend to change significantly as they adapt to local conditions and contexts or progress through scaling up. Often, those conducting the intervention develop strategies to assist schools in understanding and implementing the intervention. Despite these measures, implementation tends to vary across sites, and the outcomes—in terms of the desired change—also vary considerably. This variation in outcomes occurs largely because of the number of players involved and the number of factors that need to be in alignment to support such fundamental change.
The factors that could be expected to affect both implementation and outcomes include the following, some of which were not readily controlled by NAS or its design teams:
Here we summarize the specific findings of the various analyses. The findings are grouped into three broad areas: (1) the evolution of designs, (2) the level of implementation observed in the NAS sites and the factors related to implementation, and (3) student outcomes.
These findings are based on a continuous tracking of the designs and design teams throughout the analytic effort (Bodilly, 2001). Consistent with the literature on external change agents, the designs adapted to local conditions over time. Some of the design developments were beneficial for enabling schools to improve. For example, the growth in assistance packages, the further development of curricular units, and the development of protocols for a school's choice of design all appeared to be positive adaptations of the designs. Development of basic skills curriculum could also be considered a positive adaptation when well integrated with the principles of the design and not simply a quick add-on to meet district demands.
Other changes to the designs, while understandable, became problematic. Adaptation to district and school policies led to some designs accepting an unaligned and incoherent mix of standards, assessments, curriculum, instruction, and professional development. This situation also allowed for considerable local adaptation to the point that one might question the unifying or coherent nature of a design.
Because changes were occurring to some designs over the demonstration and scale-up phases, a major component of NAS's theory of action—a coherent and unifying design—was often missing or was constantly undergoing revision. It cannot be emphasized enough that during the entire period of the RAND studies, the designs were still in a state of development and evolution. Consistent with the literature, implementation assistance (that is, design-based assistance) became an important part of the intervention as time went on.
RAND studied the implementation of designs in both the demonstration and scale-up phases. The purpose of the studies was to measure the level of implementation in schools and to determine the conditions under which implementation prospered (Bodilly, 1998; Keltner, 1998; Bodilly and Berends, 1999; Berends, 2000; Berends et al., 2001; Kirby et al., 2001; and Berends, Bodilly, and Kirby, 2002).
The demonstration schools showed some promise, but RAND identified many barriers to implementation (Bodilly et al., 1995; Mitchell, 1996). During scale-up, NAS and the design teams partnered with schools and districts that were characterized by a host of problems related to poverty, achievement, and school and district characteristics (Berends, 1999). Achieving high levels of implementation within those schools across all teachers proved to be challenging. The case study analyses found that by two years into implementation approximately half of the sample sites were implementing designs at a level consistent with the expectations of NAS and the design teams. The other half were implementing at a rate below that level. All sites reported many barriers to further implementation.
The longitudinal sample of teachers supported the findings of the case studies that showed weak implementation and stagnation:
As expected, many factors influenced the level of implementation, and the various RAND analyses identified similar conditions that fostered higher levels of implementation.
Designs and Design-Based Assistance. Implementation varied by design. In the longitudinal survey sample, schools with CON, NARE, and RW designs clearly reported higher implementation levels. The high levels of implementation reported in all the studies were related to clear communication with and strong assistance from the design teams. In the last of the case study analyses, which used a small sample of schools, we found that several design teams did not have a large amount of information about their schools nor had they adequately measured the level of implementation at the schools. Taken together, the evidence shows that several design teams need to make significant improvements to ensure the implementation of their designs.
Selection Process. A case study in Bodilly (1998) revealed that those school administrators and teachers who felt that their choices were well informed reported higher levels of implementation at their schools than those who reported either being forced to accept a design or not understanding the nature of the design. In the longitudinal survey sample, teacher support and buy-in were related to higher levels of implementation and vice versa. The case study work in Bodilly (1998) also revealed that school principals played an important role in ensuring both a sound selection process and teacher buy-in.
School Capacity. School capacity was important in supporting implementation. The leadership of a principal was a key contributor to implementation across all the studies. The survey analyses indicated that teachers viewed a principal's leadership as the most important predictor of the implementation level achieved. The teacher survey also indicated that teachers' perceptions of students and their readiness to learn were both significantly related to the level of implementation. The various implementation case studies indicated that many other reforms were taking place in the schools while the NAS reform was under way. In many instances, the numerous reforms overloaded teachers and reduced their capacity to implement the designs.
School Context. The teacher surveys revealed that implementation was higher in high-poverty schools and schools serving large numbers of minority students. However, in schools that served significant numbers of both poor and minority students, implementation levels were significantly lower. Across our implementation studies, we found that implementation was higher in elementary schools than in secondary schools and was higher in smaller schools than in larger ones.
District Context. There were large differences in implementation among jurisdictions across all of our studies. In general, implementation was higher in those districts that (1) were more supportive of the NAS designs and were characterized as having a stable district leadership that backed the reform and made it central to its improvement efforts; (2) were not burdened with crises such as budget shortfalls or redistricting; (3) had a coherent program of reform; (4) had resources dedicated to the implementation effort; (5) had significant school-level autonomy; and (6) had trusting relationships among school, district, and union staff.
Several case study analyses highlighted significant barriers to implementation that were inherent in district and union policies. The San Antonio classroom study gave particular insight into how district-level behaviors affected schools and their willingness to undertake and implement designs (Berends et al., 2002). Teachers' views of the consistency and effectiveness of district leadership proved to be positively associated with the teachers' efforts toward implementation. However, multiple reforms easily overwhelmed teachers in their efforts at adopting a new school design. Most disruptive was the high-stakes state testing regime, which inadvertently encouraged a specific focus on basic skills and which resulted in the district adopting specific curricular programs in addition to implementing the designs. Unfortunately, these curricular programs conflicted with the designs and resulted in lower levels of implementation.
Related RAND research and exit interviews with the principals of schools that had dropped the NAS designs highlighted the importance of adequate funding. Lack of funding was the single most important reason cited by most schools in their decision to drop a design. The costs associated with the reform were far greater than the fees paid to design teams. In fact, those fees made up less than one-third of the schools' implementation costs. Significant costs were also borne by teachers in terms of their time and effort in implementing these reforms.
Our analysis of performance trends across the set of schools three years into scale-up focused on whether NAS schools made any gains in test scores relative to their respective jurisdictions. While these school-level measures allowed us to compare performance in NAS schools with that of the district as a whole, they are subject to significant limitations. For example, the aggregated measures may fail to capture changes in the tails of a distribution or may miss some significant achievement effects that would be captured if longitudinal student-level data were available and comparable across jurisdictions. In terms of student outcomes, we found the following:
Because of the wide variation in implementation that occurred and the variety of environments that exist within schools and among jurisdictions, it may have been too early to expect robust performance results across the NAS sites. Better and longer-term performance data are needed in order to make conclusive judgments about the various designs and their effects on school performance.
However, our implementation analysis showed that the overall level of implementation increased modestly over time, and there was continuing within-school variation in implementation. If the NAS approach to school improvement is correct, limited implementation will lead to weak effects on student performance. Our findings suggest that we cannot expect to have stronger performance results unless implementation significantly increases.
The detailed classroom study of the San Antonio jurisdiction allowed us to examine whether variation in instructional conditions was related to student achievement, when controlling for other student, teacher, classroom, and school characteristics. Our findings were similar to those of other analyses of schoolwide reforms:
In the end, the set of RAND analyses illuminated three important areas. First, the analyses provided evidence that an external agent such as NAS could create and promote design teams. Second, the analyses indicated that some of the theory of action behind the NAS efforts was under-developed, and they pointed to important conditions for implementation and improved student outcomes. Third, the analyses offered important lessons on how to carry out future efforts at reform.
NAS did accomplish several of the goals it had set for itself and in the process made a number of important contributions to educational reform that should be kept in mind. These include:
The RAND findings provide mixed evidence to support NAS's theories of positively changing schools with whole-school designs.
Based on our experience with NAS, we offer the following observations for any future attempts at scaling up school reforms.
Berends, Mark. (1999). Assessing the Progress of New American Schools: A Status Report. Santa Monica, CA: RAND (MR-1085-ED).
Berends, Mark. (2000). "Teacher-Reported Effects of New American Schools' Designs: Exploring Relationships to Teacher Background and School Context," in Educational Evaluation and Policy Analysis, 22(1), pp. 65–82.
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (2002). Facing the Challenges of Whole-School Reform: New American Schools After a Decade. Santa Monica, CA: RAND (MR-1498-EDU).
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (Forthcoming [b]). "District and School Leadership for Whole-School Reform: The Experience of New American Schools." In Leadership for School Reform: Lessons from Comprehensive School Reform Designs, edited by J. Murphy and A. Datnow, Thousand Oaks, CA: Corwin Press.
Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby. (Forthcoming ([a]). "Reforming Whole Schools: Challenges and Complexities." In Bringing Equity Back, edited by J. Petrovich and A. W. Wells. (n.p.).
Berends, Mark, JoAn Chun, Gina Schuyler, Sue Stockly, and R. J. Briggs. (2002). Challenges of Conflicting School Reforms: Effects of New American Schools in a High-Poverty District. Santa Monica, CA: RAND (MR-1483-EDU).
Berends, Mark, Sheila Nataraj Kirby, Scott Naftel, and Christopher McKelvey. (2001). Implementation and Performance in New American Schools: Three Years into Scale-Up. Santa Monica, CA: RAND (MR-1145-EDU).
Bodilly, Susan J. (1996a). Lessons from New American Schools Development Corporation's Demonstration Phase. Santa Monica, CA: RAND (MR-729-NASDC).
Bodilly, Susan J. (1996b). "Lessons Learned from RAND's Formative Assessment of NASDC's Phase 2 Demonstration Effort." pp. 289–324 in Bold Plans for School Restructuring: The New American Schools Designs, edited by Sam Stringfield, Steven Ross, and Lana Smith. Mahwah, NJ: Lawrence Erlbaum Associates.
Bodilly, Susan J. (1998). Lessons from New American Schools' Scale-Up Phase: Prospects for Bringing Designs to Multiple Schools. Santa Monica, CA: RAND (MR-942-NAS).
Bodilly, Susan J. (2001). New American Schools' Concept of Break the Mold Designs: How Designs Evolved and Why. Santa Monica, CA: RAND (MR-1288-NAS).
Bodilly, Susan J., and Mark Berends. (1999). "Necessary District Support for Comprehensive School Reform." pp. 111–119 in Hard Work for Good Schools: Facts Not Fads in Title I Reform, edited by Gary Orfield and Elizabeth H. DeBray. Boston, MA: Civil Rights Project, Harvard University.
Bodilly, Susan J. and Thomas K. Glennan. (1998). Reforming America's Schools: Observations on Implementing "Whole School Designs." Santa Monica, CA: RAND (RB-8016-EDU).
Bodilly, Susan J., Susanna Purnell, Kimberly Ramsey, and Christina Smith. (1995). Designing New American Schools: Baseline Observations on Nine Design Teams. Santa Monica, CA: RAND (MR-598-NASDC).
Chun, JoAn, Brian Gill, and Jodi Heilbrunn. (2001). The Relationship Between Implementation and Achievement: Case Studies of New American Schools. Santa Monica, CA: RAND (DRU-2562-EDU).
Glennan, Thomas K. Jr. (1998). New American Schools After Six Years. Santa Monica, CA: RAND (MR-945-NASDC).
Keltner, Brent R. (1998). Funding Comprehensive School Reform. Santa Monica, CA: RAND (IP-175-EDU).
Kirby, Sheila Nataraj, Mark Berends, and Scott Naftel. (2001). Implementation in a Longitudinal Sample of New American Schools: Four Years into Scale-Up. Santa Monica, CA: RAND (MR-1413-EDU).
Mitchell, Karen. (1996). Reforming and Conforming: NASDC Principals Discuss School Accountability Systems. Santa Monica, CA: RAND (MR-716-NASDC).