Jun 6, 2016
Photo by Christopher Futcher
In school year 2009–2010, the Bill & Melinda Gates Foundation launched the Intensive Partnerships for Effective Teaching initiative, a $290 million project aimed at improving student achievement through more-effective management and support of the teacher workforce.
The foundation identified seven Intensive Partnership sites—three school districts and four charter management organizations (CMOs)—to implement reforms covering teacher evaluation, staffing, professional development, and compensation and career ladders over a six-year period. These reforms are intended, among other things, to improve teachers' overall effectiveness and to ensure that low-income minority (LIM) students have access to highly effective teachers. A detailed description of the initiative is available in a RAND report, Implementation: The Intensive Partnerships for Effective Teaching Through 2013–2014..
As part of an evaluation of the Intensive Partnerships initiative, RAND Corporation researchers investigated the relationship between teachers' effectiveness in raising student achievement in mathematics and reading (the teachers' value added) and the demographic characteristics of the students the teachers serve. We measured whether, on average, LIM students were taught by more or less effective teachers than non-LIM students were. We examined the issue among teachers and students in grades 4 through 8 in four sites (Hillsborough County Public Schools in Florida; Pittsburgh Public Schools in Pennsylvania; Memphis City Schools in Tennessee; and Aspire Public Schools, a California CMO) and investigated whether site policies designed to improve LIM students' access to effective teachers have worked. We also explored differences in student access to effective teachers between schools and differences in access between classes within schools. We focused the analyses on the three years prior to the initiative's implementation and the four years following implementation.
In Hillsborough, the largest site in the study, LIM students and non-LIM students have had roughly equal access to effective teachers, and this pattern has not changed much over time. In the most recent year, LIM students’ access to effective teachers worsened somewhat: In 2014, a teacher with 10 percentage points more LIM students than other teachers had was estimated to produce 1.1 percent of a standard deviation less achievement than those other teachers in math and 0.3 percent of a standard deviation less in reading—small but statistically significant differences.
Roughly equal access in previous years, then small drop in 2014
In Memphisa, LIM students' access to effective teachers increased in 2014 after trending downward in the previous years. In 2014, a teacher with 10 percentage points more LIM students than other teachers had was estimated to produce 4.2 percent of a standard deviation better achievement than those other teachers in math and 0.9 percent of a standard deviation more in reading, both statistically significant.
a Memphis City Schools merged with Shelby County Schools shortly before the last year of the analysis period for this research.
Downward trend in previous years, increase in access in 2014
In Pittsburgh, LIM students have had greater access to effective teachers in most years in math, and we also observed this pattern in some years in reading. In 2014, a teacher with 10 percentage points more LIM students than other teachers had was estimated to produce 0.8 percent of a standard deviation better achievement than those other teachers in math and 0.9 percent of a standard deviation more in reading, with neither statistically significant.
Greater access most years in math; some years in reading
At Aspire, the only CMO in the study that was large enough to analyze student access to effective teachers, LIM students' access was about the same as that of non-LIM students in 2013–2014, after decreasing between 2009–2010 and 2011–2012. In 2014, a teacher with 10 percentage points more LIM students than other teachers had was estimated to produce virtually the same achievement as those other teachers in math and 4.2 percent of a standard deviation better achievement in reading, the latter statistically significant.
Decreased access in earlier years; roughly equal access in 2014
Effects might be expected to grow as the components are implemented more completely and transform practice more deeply.
Using student achievement data from school years 2007–2008 through 2013–2014, we constructed a value-added measure of teacher effectiveness, which we used to examine how teacher effectiveness was distributed between LIM and non-LIM students each year. In the school years prior to the initiative's implementation, teachers with more LIM students were more effective, on average, than teachers with fewer LIM students. This pattern has remained fairly consistent from 2007–2008 (several years before the intervention began) to 2013–2014, although districts differ in terms of access (see next page).
Photo by Monkey Business Images
During both the preintervention and intervention periods, effective teachers were generally more likely to be assigned to schools with higher proportions of LIM students than other schools, but, within a school, effective teachers were generally less likely to be assigned to classes with higher proportions of LIM students than to other classes. That is, the sites have been somewhat successful at placing the most-effective teachers in schools with high percentages of LIM students; however, they have been less successful in placing the most-effective teachers within each school in high-LIM classrooms.
One possible explanation for the within-school pattern could be the division of classes into advanced and regular tracks that occurs in many middle schools. However, that explanation does not appear to be true in the Intensive Partnership sites. The assignment of effective teachers to classes within elementary schools did not appear to be more favorable for LIM students than the assignment within middle schools in either reading or math in any of the districts. These findings suggest that efforts to improve LIM students' access to effective teachers should consider improving within-school access to effective teachers at all grade levels.
These findings suggest that sites' efforts to improve LIM students' access to effective teachers should focus on improving within-school access to effective teachers at all grade levels
Once sites had access to teacher effectiveness ratings, the initiative could have increased LIM students' access to effective teachers in any of three ways: by reassigning LIM students to more-effective teachers, by replacing ineffective teachers with effective teachers so LIM students have better teachers, or by improving the effectiveness of teachers who teach many LIM students. We did not find evidence that sites had accomplished any of these changes on a broad scale by 2013–2014, the first year they all had opportunities to use effectiveness ratings in teacher assignments. A few sites improved LIM students' access to effective teachers in either math or reading in one or another year, but, in most cases, improvements in one area were offset by declines in another.
Photo by Christopher Futcher/iStock
A few sites improved LIM students' access to effective teachers in either math or reading in one or another year, but, in most cases, improvements in one area were offset by declines in another
In addition to measuring LIM students' access based on our value-added measure of effectiveness, we also assessed access using the sites' own achievement growth and composite measures of teacher effectiveness. Each site calculates a composite measure that is a weighted combination of classroom-observation ratings, site-generated value-added estimates or student growth percentiles, and, in some sites, parent and student surveys. LIM students had less access to effective teachers according to the sites' composite measures of effectiveness than they did using our achievement growth measure. Furthermore, the sites' composite measures of effectiveness also showed less access for LIM students than the sites' own achievement growth measures, although the patterns varied by site and subject. However, the sites' measures showed the same pattern as our measures for changes in access to effective teachers over time and the between-school-versus-within-school split in access to effective teaching.
A possible explanation for the discrepancies between the sorting estimates based on our effectiveness measures and those based on the sites' composite measures is that the classroom-observation scores included in the composite measures do not systematically take student background into account. Although the value-added measures attempt to isolate teachers' effects by controlling for student background characteristics, the observation ratings do not. If the sites' composite measures of effectiveness are more accurate than our value-added measures, LIM students have less access to effective teaching than reported above. If, however, value added is a less biased measure, use of sites' composite measures could overstate the problem.
Given that LIM students' access to high-performing teachers has not increased significantly in most sites from preintervention levels, sites might need to expand their efforts to improve the distribution of effective teachers. They can do this through replacement, improvement, or reassignment of teachers in ways that favor LIM students. Also, the fact that the assignment of effective teachers to schools appears to be more favorable for LIM students than the assignment of effective teachers to classes within schools might suggest that the Intensive Partnership sites should pay particular attention to within-school scheduling, counseling, and curricular factors that could restrict LIM students' access to the top teachers in their schools. These sorts of changes could take longer to implement than some of the other components of the reform. Future RAND/American Institutes for Research evaluations will describe the continued implementation of the reforms through 2015–2016 and monitor changes in LIM students' access to effective teaching as the sites have more time to use effectiveness ratings in this manner.