Closing schools is never an easy or pleasant decision. Parents understandably want their local schools to stay open, as do educators and neighborhood advocates. School boards around the nation have often chosen which schools to close based on the strength of community opposition to closures. If closing School A sparks more protest than closing School B, School B gets closed.
But Pittsburgh Public Schools Superintendent Mark Roosevelt wanted to put the interests of Pittsburgh's children first. He determined that decisions about school closings should consider, above all, their effect on the achievement of Pittsburgh's students. He wants a district that is "right-sized" so that its resources are used most effectively in raising student achievement. The superintendent therefore turned to the Rand Corp. to obtain information about the performance of individual schools across the district.
Making use of a comprehensive electronic data system developed by the Pittsburgh Public Schools, Rand analyzed Pittsburgh's student achievement data, creating a new School Performance Index on a scale from one to four. Results for each school were published in a Rand report and in the Post-Gazette. (The full report is available at www.rand.org/pubs/working_papers/WR315-1.html)
Parents and educators may be wondering what the School Performance Index (SPI) means, and why the district did not use existing, publicly available measures. The proportion of students achieving proficiency on state exams, for example, is the measure used for accountability purposes under the federal No Child Left Behind Act, and it has also been used by the Allegheny Conference in assigning grades to schools across the region.
Although the proportion of students achieving proficiency is an important measure of achievement in a school, it does not distinguish the education provided by the school from the education a child receives outside of school, from family, peers and community.
Superintendent Roosevelt's aim is to close schools that are not effective at raising the achievement of the students they serve, regardless of the educational advantages that those students bring to school with them. He therefore needs a measure that attempts to identify each school's contribution to increasing the achievement of its students. Without such a measure, the district might inadvertently close schools that are effectively raising the achievement of the students they serve, sending students to schools that have higher average proficiency results but that will be less effective in raising the achievement of those students.
No statistical analysis can perfectly assess a school's contribution to student achievement, but a variety of methods, when applied carefully to high-quality achievement data, can provide much better information about school effects than is available from a snapshot of the proportion of students achieving proficiency. Rand used three different methods of analyzing school effects — each of which has advantages and disadvantages — combining them to create an SPI that is more robust than any single analytic method.
These methods examine achievement results for individual students over time, to assess their achievement gains and to examine how their achievement varies when they attend different schools. We also use information on student background characteristics — including poverty, special education status and family structure, among others — to account for some of the differences in the out-of-school academic resources of students who enroll at different schools in Pittsburgh. The aim of each of these methods, combined into the four-point SPI, is to fairly estimate the school's contribution to student achievement.
Schools that earn an SPI of four are among the strongest in Pittsburgh at raising the achievement of the students they enroll, according to our best estimate. The educators in Pittsburgh's 13 "four-star" schools, which encompass all grade levels and regions of the city, deserve praise.
SPI results do not always correspond to average proficiency results or to assessments of "adequate yearly progress" under No Child Left Behind. Pittsburgh has some high-performing schools that have not been previously recognized as such. Proficiency levels at these schools are not at the top of the scale, but the schools are serving disadvantaged students who are making larger gains than those of other students in the district.
Arsenal Middle School, for example, took a group of students who entered the school with only 14 percent achieving proficiency in math and raised their proficiency rate to 44 percent by the time they finished. Conversely, Pittsburgh also has a few schools with high levels of achievement and disappointing performance. These schools are serving relatively advantaged students who, Rand's analyses suggest, should be doing even better.
Although the SPI was created for the limited purpose of informing urgent decisions about school closings, it will undoubtedly be used by educators and parents for other purposes as well. With additional evaluation and development, the analytic methods used for the SPI may serve a variety of purposes in the future.
Rand is now working with the Pittsburgh city schools to develop a new school accountability system, compatible with federal requirements, which will examine the annual achievement growth of students in each school as one factor for determining "adequate yearly progress" under NCLB.
Still, the SPI results should be interpreted cautiously. No statistical method can unequivocally distinguish school effects from the effects of families, peers and communities. Moreover, the SPI measures only reading and math results. It doesn't assess the extent to which students are learning science, the arts, and civic skills.
Parents who are examining SPI results should keep these limitations in mind. There is no substitute for visiting schools and classrooms and talking with teachers, principals and other parents. We encourage parents to consider a wide variety of information in their assessment of the suitability of the school for their own child.
Despite these limitations, the SPI represents an important input to the district's realignment process and a key first step in larger efforts to improve instructional performance and student achievement across Pittsburgh. Challenges remain ahead. To reap an academic benefit from closing low-performing schools, the district will have to ensure that schools receiving new students have the capacity and the staff to perform better than the closed schools.
In the longer term, the development of the SPI points toward more ambitious analyses that will identify the key features of Pittsburgh's high-performing schools, laying the groundwork for system-wide improvement of school performance.
Brian Gill is a social scientist in the Pittsburgh office of the Rand Corp., a nonprofit research organization
