Evaluating Efforts to Improve School Leadership

State and school district policymakers and nonprofit organizations are increasingly focused on the critical role that principals and other school leaders play in the effectiveness of schools and in student learning. Many are targeting improvement efforts at school leadership as a way to improve student outcomes. Given the focus on accountability in education, they also are interested in evaluating whether these efforts show results. Gains in student achievement, as indicated by multiple performance measures, are the key criteria.

The nonprofit organization New Leaders is dedicated to promoting student achievement by developing outstanding school leaders to serve in urban schools. New Leaders developed a program to recruit, train, and support school leaders. Researchers from RAND Education, a unit of the RAND Corporation, have been evaluating the New Leaders program over the past five years and measuring the program's impact on student achievement. Because New Leaders has developed principals to serve in multiple states and in both charter and traditional public schools over more than a decade, the researchers have confronted and addressed many evaluation challenges. The lessons learned will be useful for other evaluators of efforts to improve school leadership and for policymakers deciding how to use student outcomes to measure an effort's impact. The challenges and recommendations highlighted here focus on evaluations of improvement efforts rather than evaluations of individual principals, although some may be relevant to both.

Allow Time for Improvement Efforts to Show

Improvement efforts that target school leadership typically affect student outcomes through the principal's influence on teachers. This indirect relationship between principals and students may cause a delay between the start of the intervention and sizable changes in student outcomes. Stakeholders will want to consider other outcomes that are available throughout the effort's lifecycle, such as changes in teacher practices, even when an effort is comprehensive enough to support a rigorous outcomes-based evaluation.

Use Multiple Measures to Evaluate Improvement Efforts

The development of metrics for assessing the success of school leadership improvement efforts poses important challenges related to the following aspects of measurement:

• Student outcome measures. Available measures typically include students' scores on state or district tests and such student-level information as attendance and whether a student graduated or progressed to the next grade. However, not all states, districts, and charter schools track the same measures in the same way, and these data have limitations that make it difficult to compare among districts.
• Controlling for student characteristics. The use of student-level data allows evaluators to control for factors that affect achievement and thus to better attribute measured changes in student outcomes to the improvement effort being evaluated. Even so, evaluation models do not control for all of the potentially relevant factors. Information known to be relevant to achievement, such as family background and composition, is not typically collected, and information that is collected, such as free and reduced-price meal status, may not measure the characteristics of interest.
• The effects of school context on student outcomes. Ideally, evaluators would compare two groups of principals in which the only difference was that one group was part of the improvement effort and the other was not. However, school context, such as teacher capacity, the size of the school leadership team, or a school's student achievement trajectory, may influence the actions that a principal takes to improve student outcomes, as well as the time it takes to realize gains. The fact that principals who participate in alternative training programs like New Leaders are often placed in atypical or unique school settings makes a simple comparison of the outcomes uninformative because it is likely that the difference in outcomes stems from various factors.
• Accounting for principal characteristics. Prior research suggests that it is particularly important to account for the level of experience a principal has leading a school, other leadership experience, and experience in the same school prior to becoming a principal. Principals bring a wide range of backgrounds to their schools, making it difficult to isolate the effect of the improvement effort.

The use of multiple qualitative and quantitative measures can help to overcome the challenges that evaluators and policymakers face by providing a more well-rounded evaluation of the school leadership effort's effect. The choice of measures should be guided by the specific pathway through which a policy, practice, or program seeks to influence school leadership and student outcomes.

Choose Suitable Evaluation Methods and Interpret Results Appropriately

Finally, both evaluators and policymakers can contribute to careful planning at the beginning of an evaluation and careful interpretation of findings as the evaluation progresses.

Recommendations for Evaluators

As much as possible, evaluators should weigh carefully the decision of which outcome measures and results to emphasize over others. For example, evaluators should use caution with district-created variables and be sure they understand how these are created, especially if the effort covers multiple districts or charter school management organizations. If possible, evaluators should obtain detailed entry and withdrawal codes from the district so that they can differentiate among student mobility, graduation, and dropout. The evaluator can then construct variables that reflect these different exit pathways.

Evaluators should consider the size and structure of a school's leadership team and the degree of influence the principal has over that team when evaluating efforts to improve principal effectiveness. Because districts provide differing levels of principal autonomy over staffing, this influence is likely to vary by district.

Recommendations for Policymakers

Policymakers have a role to play in mitigating the challenges that evaluators face. They should lay the groundwork for the collection and retention of data that will be needed for the purposes of evaluation at the start of an initiative. There also may be a role for state policymakers to encourage consistent and systematic reporting of core variables of interest by all districts. And developing a data system that allows for more rapid updating of district data would allow the inclusion of startup schools in evaluations and would better reflect changing school composition.

Policymakers are strongly cautioned that, when assessing the effects of efforts targeting principals, it is important to understand and question the methods used to create a comparison group for evaluation purposes and to consider that there may be underlying reasons why one group may differ vastly from another.

As context and grouping of schools matter, policymakers should assess the extent to which the results of other studies are generalizable to their school, district, or state. In adopting an improvement effort that has been effective in another context, policymakers should pay particular attention to whether the new context has similar features and consider the potential implications of any differences.

Topics

• Academic Achievement
• Education Policy
• Principals
• Teacher Incentives