Sep 21, 2020
Photo by New Africa/Adobe Stock
Research shows that effective teachers are one of the most important in-school contributors to student learning. To improve low-income minority students' access to effective teaching, the Bill & Melinda Gates Foundation launched the Intensive Partnerships for Effective Teaching (IP) initiative in the 2009–2010 school year. The program was implemented in three public school districts and four charter management organizations (CMOs) across the United States.
Researchers from the RAND Corporation and the American Institutes for Research (AIR) assisted the Gates Foundation in understanding the impact of the initiative, which ended in 2016. The researchers examined a variety of outcomes, but analysis suggested that, overall, the initiative did not reach its goals. In particular, student achievement, low-income minority students' access to effective teachers, and dropout rates were not substantially better in the IP sites than in similar sites that did not participate in the initiative.
Although this analysis of impact, which was published in 2018, did not find that the initiative was effective overall, the researchers did observe that some IP schools had improved more than others. The Gates Foundation and the study team decided to dig deeper into the variation in outcomes among schools and look for school-level factors associated with that variation. The three districts and one of the CMOs that had participated in the IP initiative agreed to participate in further data collection. Newly collected qualitative data, as well as new analyses of survey data from the original study, allowed the team to more closely investigate the factors that differed between the improving and nonimproving IP schools and develop recommendations for future variation studies of multisite efforts.
The research team interviewed staff and leadership from pairs of schools in Hillsborough County Public Schools in Florida, Pittsburgh Public Schools in Pennsylvania, Shelby County Schools in Tennessee, and Aspire Public Schools in California. The two schools in each pair were similar in terms of student achievement levels at the outset of the initiative but differed in the extent to which they had improved during the initiative. Team members visited the school pairs in spring 2018 to gather staff perspectives on implementation of the initiative and on several factors that were potentially relevant to their schools' improvement. The team members were unaware of which school in each pair was the improver and which was the nonimprover during both their visits and much of the subsequent data analysis. From these interviews, several factors emerged as associated with schools' improvement or nonimprovement.
Two leadership behaviors were associated with IP school improvement or nonimprovement. First, interviewees at improving schools were more likely than those at nonimproving schools to report that their school leaders encouraged collaboration and learning among staff members. Collaboration was facilitated in several ways, including by creating time for teacher collaboration and asking teachers with specific skills to assist with the professional development of other teachers. Second, principals at improving schools gave teachers a greater degree of decisionmaking power. Specifically, teachers at improving schools were more likely to report that they felt trusted by their school leaders to make decisions about their own classrooms and that principals actively sought their input for school-level decisions.
Perceptions of staff cohesion appeared to be related to school improvement status in two ways. First, interviewees at improving schools reported broad support for a common mission more often than those in nonimproving schools. Second, it was more common for interviewees in nonimproving schools to discuss divisiveness among staff members than it was in improving schools. Furthermore, interviewees indicated that school leaders could contribute to the sense of cohesion or divisiveness among staff at the schools.
Interviewees at nonimproving schools were more likely to report that influences outside their control shaped their schools' success or lack thereof. These interviewees described three specific outside factors: (1) high-achieving students were "siphoned" by other schools in the area; (2) that student or community demographic factors, such as socioeconomic status or ethnicity, were associated with their schools' potential for achievement; and (3) student discipline policies, which typically are set at the district level, limited their options to manage student behavior. Notably, our data do not confirm that demographic shifts in schools were associated with nonimprovement. Therefore, interviewees' reports might reflect their perceptions or beliefs rather than actual changes.
As another approach to exploring variation among schools, the research team conducted new analyses of data from surveys that were administered to teachers each year from 2013 through 2016 as part of the main evaluation of the IP initiative. These analyses sought to determine whether particular constructs measured by the surveys, aggregated to the school level, were associated with school improvement.
Of the 12 factors measured by the surveys, the following seven were found to be significantly related to school improvement:
For example, schools in which teachers expressed higher levels of support for their districts' (or CMO's) evaluation system—and, in particular, the system's student achievement component—experienced greater improvement than schools in which teachers were less supportive of the system. The alignment and relevance of professional development, support for the evaluation system, and validity of the student achievement component factors had the largest overall effects. For all seven, there were no significant differences among the four IP locations included in the analysis. However, the results do not necessarily mean that more-positive attitudes caused greater school improvement. It is possible that causality might have operated in the reverse direction; that is, that school improvement fostered more-positive attitudes.
Together, the interview and survey portions of this study suggest that schools in which leaders provided decisionmaking opportunities to teachers and in which teachers reported a sense of unity were more likely to have improved. Notably, factors external to the schools (e.g., changes in state policies) were not dimensions of the IP initiative; rather, they were contextual factors at individual schools. The survey portion of the study found that participant perceptions of some aspects of the IP initiative were associated with improvement, but the interview portion did not find that to be the case.
Because the two portions of the study differed in many respects—purpose, methodological approach, samples, time frames, and measures—it is not surprising that there were some differences in their findings. A limitation of both was that analyses were correlational rather than causal; another is that the study was conducted after the initiative had ended.
The research team drew on its experience to suggest recommendations for studying variation in outcomes across schools implementing a large-scale intervention like the IP initiative. To study variation in outcomes in similar studies, researchers should
This study complements the multiyear evaluation of the Gates Foundation's Intensive Partnerships for Effective Teaching (IP) initiative. The evaluation was conducted by researchers at the RAND Corporation and the American Institutes for Research. Using retrospective qualitative data collection and analysis, the researchers find associations between school improvement (or nonimprovement) and staff perceptions of school leaders' behaviors, staff cohesion, and factors outside schools' control. Additional survey analyses indicate that perceptions of district and school climate, professional development, teacher evaluation systems and components implemented as part of the IP initiative, and compensation were associated with school improvement.