The Intensive Partnerships for Effective Teaching

Photo by New Africa/Adobe Stock

Research shows that effective teachers are one of the most important in-school contributors to student learning. To improve low-income minority students' access to effective teaching, the Bill & Melinda Gates Foundation launched the Intensive Partnerships for Effective Teaching (IP) initiative in the 2009–2010 school year. The program was implemented in three public school districts and four charter management organizations (CMOs) across the United States.

Researchers from the RAND Corporation and the American Institutes for Research (AIR) assisted the Gates Foundation in understanding the impact of the initiative, which ended in 2016. The researchers examined a variety of outcomes, but analysis suggested that, overall, the initiative did not reach its goals. In particular, student achievement, low-income minority students' access to effective teachers, and dropout rates were not substantially better in the IP sites than in similar sites that did not participate in the initiative.

Although this analysis of impact, which was published in 2018, did not find that the initiative was effective overall, the researchers did observe that some IP schools had improved more than others. The Gates Foundation and the study team decided to dig deeper into the variation in outcomes among schools and look for school-level factors associated with that variation. The three districts and one of the CMOs that had participated in the IP initiative agreed to participate in further data collection. Newly collected qualitative data, as well as new analyses of survey data from the original study, allowed the team to more closely investigate the factors that differed between the improving and nonimproving IP schools and develop recommendations for future variation studies of multisite efforts.

According to Interviews, School Improvement Was Associated with Specific Principal Behaviors, Staff Cohesion, and Interviewee Perceptions of Factors Outside the Schools' Control

The research team interviewed staff and leadership from pairs of schools in Hillsborough County Public Schools in Florida, Pittsburgh Public Schools in Pennsylvania, Shelby County Schools in Tennessee, and Aspire Public Schools in California. The two schools in each pair were similar in terms of student achievement levels at the outset of the initiative but differed in the extent to which they had improved during the initiative. Team members visited the school pairs in spring 2018 to gather staff perspectives on implementation of the initiative and on several factors that were potentially relevant to their schools' improvement. The team members were unaware of which school in each pair was the improver and which was the nonimprover during both their visits and much of the subsequent data analysis. From these interviews, several factors emerged as associated with schools' improvement or nonimprovement.

Staff Perceptions of Principal Behaviors

Two leadership behaviors were associated with IP school improvement or nonimprovement. First, interviewees at improving schools were more likely than those at nonimproving schools to report that their school leaders encouraged collaboration and learning among staff members. Collaboration was facilitated in several ways, including by creating time for teacher collaboration and asking teachers with specific skills to assist with the professional development of other teachers. Second, principals at improving schools gave teachers a greater degree of decisionmaking power. Specifically, teachers at improving schools were more likely to report that they felt trusted by their school leaders to make decisions about their own classrooms and that principals actively sought their input for school-level decisions.

Staff Cohesion

Perceptions of staff cohesion appeared to be related to school improvement status in two ways. First, interviewees at improving schools reported broad support for a common mission more often than those in nonimproving schools. Second, it was more common for interviewees in nonimproving schools to discuss divisiveness among staff members than it was in improving schools. Furthermore, interviewees indicated that school leaders could contribute to the sense of cohesion or divisiveness among staff at the schools.

Outside Factors

Interviewees at nonimproving schools were more likely to report that influences outside their control shaped their schools' success or lack thereof. These interviewees described three specific outside factors: (1) high-achieving students were "siphoned" by other schools in the area; (2) that student or community demographic factors, such as socioeconomic status or ethnicity, were associated with their schools' potential for achievement; and (3) student discipline policies, which typically are set at the district level, limited their options to manage student behavior. Notably, our data do not confirm that demographic shifts in schools were associated with nonimprovement. Therefore, interviewees' reports might reflect their perceptions or beliefs rather than actual changes.

Surveys Suggested That Perceptions of District and School Climate, Professional Development, Teacher Evaluation, and Compensation Were Associated with School Improvement

As another approach to exploring variation among schools, the research team conducted new analyses of data from surveys that were administered to teachers each year from 2013 through 2016 as part of the main evaluation of the IP initiative. These analyses sought to determine whether particular constructs measured by the surveys, aggregated to the school level, were associated with school improvement.

Of the 12 factors measured by the surveys, the following seven were found to be significantly related to school improvement:

• supportive district climate
• supportive and respectful school environment
• impetus for and impact of professional development
• alignment and relevance of professional development
• support for the evaluation system
• validity of the student achievement component of the evaluation system
• fairness of compensation.

For example, schools in which teachers expressed higher levels of support for their districts' (or CMO's) evaluation system—and, in particular, the system's student achievement component—experienced greater improvement than schools in which teachers were less supportive of the system. The alignment and relevance of professional development, support for the evaluation system, and validity of the student achievement component factors had the largest overall effects. For all seven, there were no significant differences among the four IP locations included in the analysis. However, the results do not necessarily mean that more-positive attitudes caused greater school improvement. It is possible that causality might have operated in the reverse direction; that is, that school improvement fostered more-positive attitudes.

Study Limitations and Recommendations

Together, the interview and survey portions of this study suggest that schools in which leaders provided decisionmaking opportunities to teachers and in which teachers reported a sense of unity were more likely to have improved. Notably, factors external to the schools (e.g., changes in state policies) were not dimensions of the IP initiative; rather, they were contextual factors at individual schools. The survey portion of the study found that participant perceptions of some aspects of the IP initiative were associated with improvement, but the interview portion did not find that to be the case.

Because the two portions of the study differed in many respects—purpose, methodological approach, samples, time frames, and measures—it is not surprising that there were some differences in their findings. A limitation of both was that analyses were correlational rather than causal; another is that the study was conducted after the initiative had ended.

The research team drew on its experience to suggest recommendations for studying variation in outcomes across schools implementing a large-scale intervention like the IP initiative. To study variation in outcomes in similar studies, researchers should

• conduct extensive qualitative data collection during implementation to support robust examination of school-level variation. Because the IP variation study in this research was conducted after the primary IP evaluation was complete, the research team needed to interview staff at schools retrospectively. It would have been preferable to conduct such in-depth interviews during implementation. If feasible, researchers should collect variation-related qualitative data as part of initial evaluation efforts. Interviews, focus groups, classroom observations, and artifacts related to instruction and to principals' observations of and feedback to teachers could be helpful in assessing how different schools adopted and interpreted different initiative components.
• design evaluations to capture multiple levels of implementation and impact. Variation in impact can occur at multiple levels of the site, school leader, teacher, and student pyramid. The original IP evaluation focused on variation across districts and CMOs in implementation and impact. In contrast, this study focused on school-level variation. Neither study was designed to understand variation among teachers in the extent to which their effectiveness improved. Additional data would have been useful to understand variation in terms of individual teachers and school leaders' roles in this variation.

This study complements the multiyear evaluation of the Gates Foundation's Intensive Partnerships for Effective Teaching (IP) initiative. The evaluation was conducted by researchers at the RAND Corporation and the American Institutes for Research. Using retrospective qualitative data collection and analysis, the researchers find associations between school improvement (or nonimprovement) and staff perceptions of school leaders' behaviors, staff cohesion, and factors outside schools' control. Additional survey analyses indicate that perceptions of district and school climate, professional development, teacher evaluation systems and components implemented as part of the IP initiative, and compensation were associated with school improvement.

Available for Download

Download PDF

Topics

• Academic Achievement
• Minority Students
• Principals
• Teacher Effectiveness