Interim Findings on Teacher Reform Effort Show Small Positive Impact on Student Outcomes During Initial Years

A study conducted by the RAND Corporation and the American Institutes for Research examining the impact of teacher reforms in seven sites across the country shows signs of progress.

With a few exceptions, the reforms did not have significant impact on student outcomes during the initial years. However, the impact on student outcomes began to show a positive trend during the 2013–2014 school year.

The reforms are part of the Intensive Partnerships for Effective Teaching initiative launched by the Bill & Melinda Gates Foundation in 2009–2010.

The reforms are intended to improve overall effectiveness of teachers and to ensure that low-income minority students have access to highly effective teachers. The overall goal is to produce significant gains in student achievement, higher graduation rates and increase the number of college-bound students, especially among low-income minority students.

“Since each school site followed a somewhat different implementation plan and it takes time for policy changes to influence teaching practice and student learning, it is not clear when the sites should expect to see more-significant improvements in student outcomes,” said Brian Stecher, the study's lead author and a senior behavioral social scientist at RAND, a nonprofit research organization. “We have two additional years of test data to evaluate, and the sites are hopeful that the upward trend will continue.”

The Gates Foundation worked with a set of Intensive Partnerships sites — three school districts across the United States and four charter management organizations in California — to implement reforms covering teacher evaluation, staffing, professional development, compensation and career ladders over a six-year period. The foundation asked RAND and the American Institutes for Research to evaluate the initiative.

The research team notes that each school district and charter management organization had different administrative policies when the process began and therefore each took a different amount of time to implement the reforms.

Each site developed a shared vision of effective teaching, developed a classroom observation ratings scale and devised a new measure of teacher effectiveness that included classroom observations, as well as each teacher's impact on student achievement. Although stakeholders took up to two years to agree on a new definition of effective teaching and revise their evaluation systems, every site adopted the core elements, leading to a common vision and a change in the conversation about effective teaching, according to researchers.

The analysis of the three school districts — Pittsburgh, Pa., Memphis, Tenn., and Hillsborough County in Florida — indicates an upward trajectory in five of the six average achievement outcomes in grades 3–8 in the most recent year. Researchers say this suggests the reforms might be on the way to having a positive impact on student outcomes after a few transition years in which there was zero or negative impact. The majority of teachers surveyed in 2015 indicated that they used feedback from the new evaluations to make changes in their teaching practice.

“A delay in positive effects is not completely surprising, given the time it took to implement many of the initiative's elements,” said Michael S. Garet, vice president at American Institutes for Research. “These sites completely overhauled their evaluation, staffing, professional development, compensation and career policies, which does not happen overnight.”

Some effects were detected earlier. For example, Pittsburgh has seen a significant positive impact on graduation rates in most years. Hillsborough County saw statistically significant positive results in reading achievement in 2014 for economically disadvantaged students in the third through eighth grades. In Memphis, declining math and reading scores (relative to the rest of the state) have showed improvement since 2012–13, when implementation of the initiative was more complete.

As part of the analysis, researchers found that all school sites are holding on to their best performing teachers at higher rates than their lowest performing teachers. They also found each of the school organizations have been more successful at placing the most effective teachers in schools with high percentages of low-income minority students.

However, there has been less success in placing the most effective teachers within each school in individual classrooms with a high percentage of low-income minority students. This suggests that efforts to improve low-income minority students' access to effective teachers should focus on improving within-school access to effective teachers at all grade levels, according to researchers.

The evaluation is continuing and a final report expected in 2017 will incorporate two more years of student test results and other data.

Support for this research was provided by The Bill & Melinda Gates Foundation.

The reports, “Improving Teaching Effectiveness: Implementation,” “Improving Teaching Effectiveness: Impact on Students,” and “Improving Teaching Effectiveness: Access to Effective Teaching,” are available at www.rand.org.

Other authors of the study are Laura S. Hamilton, John Engberg, Elizabeth D. Steiner, Abby Robyn, Matthew Baird, Gerald Hunter, Ben Master, Italo Gutierrez and Gabriel Weinberger, all of RAND, and Jeffrey Poirier, Deborah Holtzman, Eleanor Fulbeck, Jay Chambers and Iliana Brodziak de los Reyes, all of the American Institutes for Research.

This research was conducted by RAND Education, a division of the RAND Corporation. Its mission is to bring accurate data and careful, objective analysis to the national debate on education policy.