Student Achievement in Privately Managed and District-Managed Schools in Philadelphia Since the State Takeover

Introduction

In 2002, the state of Pennsylvania, frustrated by years of low achievement and a decade of budget crises in the School District of Philadelphia, took charge of the city's 200,000-pupil system. The state replaced Philadelphia's nine- member school board with an appointed School Reform Commission (SRC) composed of three members appointed by the governor and two appointed by the city's mayor. The SRC then hired a new CEO who immediately instituted sweeping changes, including the implementation of districtwide common curricula and a system of frequent benchmark assessments to be used for diagnostic purposes. More controversially, the SRC adopted a "diverse provider" model as it turned over management of 45 of the district's lowest-performing elementary and middle schools to seven for-profit and nonprofit organizations, including two local universities; the private managers were given additional per-pupil funding to support their work. For the last four years, Philadelphia has been the site of the nation's largest experiment in the private management of public schools. Philadelphia's experience may have implications for schools and districts across the country: State takeover and private management are two of the interventions that can be applied to chronically low-achieving schools and districts under the federal No Child Left Behind Act (NCLB).

In addition to privately managed schools, the diverse provider model in Philadelphia also included two groups of low-achieving schools that were given special support and/or funding while remaining under district management. At the same time the SRC brought in the private managers, it "restructured" an additional 21 low-performing schools, providing intensive staff support and extra per-student funding, and provided 16 other schools that were perceived as improving (and became known as the "sweet 16") with increased funding but no additional intervention.

This research brief summarizes findings presented in State Takeover, School Restructuring, Private Management, and Student Achievement in Philadelphia, which analyzes achievement differentials associated with Philadelphia's privately managed, restructured, and sweet 16 schools, and examines whether different private providers had different effects. Prior to examining differences among treatments within the Philadelphia public schools, that study sets the context for the analysis by examining districtwide achievement trends in the first four years following the state takeover (through spring 2006). It does not aim to provide a rich and comprehensive assessment of all aspects of Philadelphia's school improvement efforts since 2002; it merely evaluates, to the extent possible, the achievement impacts of some of those efforts, as measured by results on annual assessments in reading and mathematics.

The Diverse Provider Model

Philadelphia's diverse provider model borrows from theoretical models by which districts promote internal competition through school choice and the establishment of clearly distinguished alternatives, but the district's implementation diverged from the theory in important ways. The Philadelphia model was characterized by little competition among providers and by the absence of parental choice among the educational models offered. In addition, continued district involvement in provider schools and mandated districtwide initiatives constrained provider autonomy. As a result, the education providers' improvement strategies looked more similar than might have been predicted. Consequently, Philadelphia's experience should not be viewed as a definitive test of private management under competitive conditions.

Districtwide Achievement Trends in Philadelphia

The proportion of elementary and middle-school students achieving proficiency in reading and math has increased substantially in Philadelphia in the years since the state takeover, according to the state achievement test (the Pennsylvania System of Student Assessment, or PSSA). From the 2001–2002 school year to the 2005–2006 school year, an additional 11 percent and 23 percent of students reached proficiency in fifth-grade reading and math, respectively. Similarly, an additional 20 percent and 19 percent of students reached proficiency in eighth-grade reading and math, respectively. (We use fifth- and eighth-grade reading and math proficiency levels because these two grades were tested consistently in both subjects across the time horizon of the study.)

Whether those improvements exceed the gains of similar schools elsewhere in Pennsylvania and whether they can be attributed to the interventions implemented by the SRC since 2002 are harder questions. We gauge the impact of the SRC's interventions by comparing the proficiency results, before and after the state takeover, of schools in Philadelphia to those of a comparison group of schools elsewhere in Pennsylvania. This is necessarily an imperfect analysis because it must rely on imprecise, school-level achievement results and because there are no other school districts in Pennsylvania quite like that in Philadelphia. The results therefore require caution in interpretation.

Although it is impossible to find perfect matches for Philadelphia schools elsewhere in Pennsylvania, we can reduce the differences between Philadelphia and the comparison schools by examining only schools in which achievement results were in the state's lowest quartile in both 2001 and 2002, immediately prior to the creation of the SRC. This includes most of Philadelphia's elementary and middle schools.

An examination of net improvements over the four-year period since the state takeover of the district shows that Philadelphia's low-quartile schools outgained low-quartile comparison schools by a statistically significant margin in eighth-grade reading. In fifth-grade reading, fifth-grade math, and eighth-grade math, the four-year gains among Philadelphia's low-quartile schools, as compared with pretakeover baseline scores, were indistinguishable from the gains of low-quartile schools elsewhere in Pennsylvania.

In sum, Philadelphia has seen substantial districtwide gains in the proportion of students achieving proficiency since the 2002 state takeover. But after four years, the gains of its low-achieving schools (constituting most of the schools in the district) have generally not exceeded the gains of low-achieving schools elsewhere in Pennsylvania.

Achievement Effects of Diverse Providers

Turning to the diverse provider model, we examine how the change in management affected the achievement of students in the privately managed, restructured, and sweet 16 schools. Each of these management models can be viewed as a "treatment," for which it is possible to examine effects on students in treated schools. We examine the achievement of students in schools under different treatments, before and after the state takeover, and compare their trends with the trends of other students in Philadelphia. This "fixed-effect" approach allows each student to serve as his or her own control, thereby factoring out characteristics of students (such as race, ethnicity, and other unchanging family and student characteristics) that may affect student achievement results. It also allows each school to serve as its own control, which is particularly important because schools were selected for treatment because they had a history of low achievement.

The major findings of the analysis of achievement effects under the diverse provider model in its first four years of operation are as follows:

• Sweet 16 schools: There were no statistically significant effects, positive or negative, in reading or math, in any of the four years in which they received additional resources.
• Privately managed schools (as a group): There were no statistically significant effects, positive or negative, in reading or math, in any of the four years after takeover.
• Restructured schools: There were significantly positive effects in math in all three years of implementation and in reading in the first year. In the fourth year, after the Office of Restructured Schools had been disbanded and the additional resources for the schools had ceased, the former restructured schools maintained a substantial (though only marginally statistically significant) effect in math.

In short, after four years of intervention, achievement gains in privately operated schools and sweet 16 schools, on average, are no different from Philadelphia's districtwide gains. Meanwhile, restructured schools outgained the rest of the district in math in all three years of restructuring, with evidence that the gain persisted a year after the conclusion of restructuring. These positive effects for restructured schools in math (ranging from 0.16 to 0.22 standard deviations) are moderate to large in size, when compared with effects seen in educational interventions generally.

When grouping the external managers as universities, other nonprofits, and for-profits, we find no statistically significant effects, positive or negative, of any of the three provider types. Results for individual providers likewise show few statistically significant effects, though two providers show cause for concern: Four-year results were significantly and substantially negative for Temple University schools in both subjects and for Victory schools in math.

We find no clear indications of any notable differences in effects on particular at-risk subgroups of students, such as special education students or those with limited English skills.

Conclusions and Implications

Different interpretations of the results may lead to different judgments about whether Philadelphia's experiment in the private management of public schools has succeeded and whether it should be continued. On the negative side of the ledger, despite additional per-pupil resources, privately operated schools did not produce average increases in student achievement that were any larger than those seen in the rest of the district. Meanwhile, district-managed restructured schools outpaced the gains of the rest of the district in math.

Nonetheless, it is impossible to know definitively how the privately managed schools would have done if they had been restructured or remained under conventional district management. Whether the district could have replicated the gains of the restructured schools in three times as many schools—as it would have needed to do if the 45 schools turned over to private management had been restructured instead—is an open question. The private managers were given some of the lowest-achieving schools in the district. Although their trajectories did not exceed those of the district, they improved alongside the rest of the district at a time when achievement levels districtwide were increasing substantially. We found little reason to believe that the districtwide improvement was a consequence of competition from private providers, but we cannot rule out the possibility that the assistance from private providers was an important part of the total reform effort in Philadelphia.

Even so, although it is theoretically possible that the introduction of the providers increased districtwide capacity for improvement, and that the schools they managed would have done worse without them, we find no evidence supporting this view. If the privately managed schools had remained under district management instead, it seems likely that the district could have replicated the gains of other schools that received no special interventions—getting results similar to those actually achieved by the private providers without expending additional resources. In sum, with four years of experience, we find no evidence of differential academic benefits that would support the additional expenditures on private managers. The private managers may also be producing other benefits that are not measurable in terms of student achievement results in math and reading.

The larger implications of these findings for the most aggressive sanctions of NCLB are less clear. With respect to state takeover, results are ambiguous: Subsequent to the state's takeover of the district, proficiency percentages increased districtwide, but the total increase over four years was not substantially greater than the increase of other low-achieving schools in the state, in most cases. Philadelphia provides no evidence to support private management as an especially effective method of promoting student achievement, but it does not represent a clear test of full private management in a competitive market. Whether a model of private management that involves more autonomy to managers, parental choice, and competition for students would produce better results remains an open question.