Trends in Impact on Student Outcomes

Photo by PeopleImages/iStock

In school year 2009–2010, the Bill & Melinda Gates Foundation launched the Intensive Partnerships for Effective Teaching initiative, a $290 million project aimed at improving student achievement through more-effective management and support of the teacher workforce.

The foundation identified seven Intensive Partnership sites — three school districts and four CMOs — to implement, over a six-year period, reforms covering teacher evaluation, staffing, professional development, and compensation and career ladders.^[1]

Hoped-for Positive Effects on Student Outcomes Not Seen

As of the 2013–2014 school year, most of the elements of the Intensive Partnerships initiative had been implemented for one or more years and had an opportunity to influence teacher placement, teacher practices, and student outcomes. However, through the 2013–2014 school year, the initiative has not had the dramatic positive effects on student outcomes for which the foundation had hoped. Most estimated effects on student outcomes were insignificant or negative. The estimated positive effects in grades 3–8 in the 2013–2014 school year in two of the three Intensive Partnership districts studied are mostly statistically nonsignificant, although these effects are about the same size as the average effects found for charter schools in several large-scale studies and for other districtwide interventions that researchers have studied. The estimated effect sizes appear small compared with the average expected achievement gains during a year of learning or with benchmarks from previous school-level interventions. Nevertheless, the initiative is still under way, and recent gains might signal future improvements.

About the Study

Photo by Yuri_Arcurs/iStock

The foundation selected the RAND Corporation and its partner, the American Institutes for Research (AIR), to evaluate the Intensive Partnership efforts. The evaluation began in July 2010, and data collection continues through the 2015–2016 school year. The RAND/AIR team is conducting three interrelated studies examining the implementation of the reforms, the reforms' impact on student outcomes, and the extent to which the reforms are replicated in other districts. The team will produce a final report in 2017.

This brief focuses on the initiative's impact on student outcomes in the three district sites: Pittsburgh Public Schools (PPS) in Pennsylvania, Hillsborough County Public Schools (HCPS) in Florida, and Memphis City Schools (MCS) in Tennessee, using state testing data as the primary measure of student achievement. The team's evaluation of outcomes includes both school-level and district-level analyses to compare student outcomes in the Intensive Partnership sites and similar non–Intensive Partnership schools and districts in the same states.

Limited Impact Is Seen in Sites, but an Uptick in Effects in School Year 2013–2014 Could Be Promising

With the exception of MCS, which fared significantly worse after the start of the initiative, most changes in student achievement in lower grades (3–8) in Intensive Partnership schools from 2009 to 2014 were not statistically different from changes in other comparable schools in those states. However, specific impacts of the Intensive Partnership intervention varied by site, grade, subject, and year of evaluation.

Although the estimated impact of the Intensive Partnerships initiative appears to be positive for student achievement in grades 3–8 in math or reading in HCPS and PPS for the 2013–2014 school year, the size of these gains is modest. Also, only the average gain for grade 3–8 math in PPS is statistically significant. Overall, the estimated effect sizes are small compared with the average expected achievement gains during a year of learning or with other benchmarks from school-level interventions (such as Comprehensive School Reform). However, the estimated effect size in grade 3–8 math in PSS is larger than the average effects found for charter schools and for some other districtwide interventions that researchers have studied. Also, the estimated effect sizes in grade 3–8 reading in HCPS and PPS, albeit not statistically significant, are about the same as those sizes found for district-level interventions and charter schools.

One interesting pattern that bears mentioning is the upward trajectory of almost all outcomes between 2012–2013 and 2013–2014. In all three districts, impact estimates in most subject-grade combinations were greater in 2014 than in 2013 as sites implemented more of the reform components. Although only the math impact estimate in grades 3–8 in PPS is positive and statistically significant for 2014, the recent upward trajectory in five of the six grade 3–8 achievement impact estimates suggests that the reforms might be on the way to having a positive impact after a few transition years with little or negative impact. This would not be surprising, given the time it took to implement many of the initiative's elements.^[2] The authors will update these estimates based on 2014–2015 and 2015–2016 test results to see whether the promising trends continue.

Difference in Gains on State Standardized Tests

Difference in standard deviation units between gains in schools in the Intensive Partnership district and gains in similar schools in the state

Hillsborough

In Hillsborough County, there is evidence that grades 3–8 have been making more progress than similar schools in the state in reading in 2014, although the estimate is not statistically significant. In grade 3–8 math, schools of HCPS performed similarly in 2014 to comparable schools in other Florida districts.

The authors found that high schools in HCPS underperformed on reading scores compared with comparable Florida high schools and that dropout rates were higher than in comparison schools.

Math, Grades 3–8

Reading, Grades 3–8

Memphis

In Memphis, although scores in math and reading in grades 3–8 worsened more than scores in similar schools in the initial years after the grant was awarded, they have improved since the 2012–2013 school year, as the initiative has been more fully implemented.

The testing policies in Tennessee prevented the authors from estimating the initiative's impact on high school student achievement, but they did find evidence of higher dropout rates than those in similar Tennessee high schools following program implementation. There was some difficulty forming an adequate comparison group of Tennessee schools with similar trends in achievement and demographics before the intervention, but these results were consistent using two different comparison methods.

Math, Grades 3–8

Reading, Grades 3–8

Pittsburgh

In Pittsburgh, there are some indications that math and reading achievement in grades 3–8 improved more than in comparable schools in Pennsylvania in 2014, but the difference was only statistically significant for math.^[3]

High schools in PPS underperformed on reading scores relative to comparable Pennsylvania high schools. There has, however, been a significant positive effect on graduation rates in most years.

Math, Grades 3–8

Reading, Grades 3–8

Conclusion

Analyses of the implementation of the Intensive Partnership reforms demonstrate that it takes time to build new systems and processes that support effective teaching and to develop and sustain stakeholder engagement for such an undertaking. In this respect, it is perhaps unsurprising that results thus far have shown limited gains. Future RAND/AIR evaluations will describe the continued implementation of the reforms through 2015–2016 and provide two more years of evidence about the effect that the Intensive Partnership interventions has had on student achievement and other outcomes.

Key Findings

The Intensive Partnerships for Effective Teaching

1. Through 2013–2014, most estimated effects on student outcomes were insignificant or negative, but there was a positive effect in lower grades (3–8) in math in Pittsburgh Public Schools. This effect is equivalent to approximately two and a half months of learning.
2. Most of the estimated effects on student outcomes are trending up in the latest year. Although most of them were not statistically significant as of the 2013–2014 school year, these trends might suggest that the gradual implementation of the initiative is beginning to yield dividends.

Notes

• [1] We use the word site to describe the three school districts and the four charter management organizations (CMOs) that received funding from the foundation to implement the Intensive Partnerships initiative. This brief about impact does not include results for any of the CMOs because student achievement data for the 2013–2014 school year are not available for California, where most of these schools are located.
• [2] For a comprehensive description of which practices each site has implemented, see Brian M. Stecher, Michael S. Garet, Laura S. Hamilton, Elizabeth D. Steiner, Abby Robyn, Jeffrey Poirier, Deborah Holtzman, Eleanor S. Fulbeck, Jay Chambers, and Iliana Brodziak de los Reyes, Implementation, Appendixes D and E: The Intensive Partnerships for Effective Teaching Through 2013–2014, Santa Monica, Calif.: RAND Corporation, RR-1295/2-BMGF, 2016.
• [3] For this brief, we define statistical significance at the 0.05 level and evaluate it based on a school-level difference-in-differences methodology. In the full report, the authors also present a synthetic-control-group comparison for the Intensive Partnership districts that the authors conducted as a robustness check on the difference-in-differences methodology.

Available for Download

Download PDF

Topics

• Academic Achievement
• Education Reform
• Human Capital Investments
• Teacher Effectiveness
• Workforce Management