Oct 4, 2012
Efforts to improve school leadership may improve student outcomes. Evaluation is critical for effective use of resources, but it poses challenges. Recommendations for evaluators and policymakers are as follows:
State and school district policymakers and nonprofit organizations are increasingly focused on the critical role that principals and other school leaders play in the effectiveness of schools and in student learning. Many are targeting improvement efforts at school leadership as a way to improve student outcomes. Given the focus on accountability in education, they also are interested in evaluating whether these efforts show results. Gains in student achievement, as indicated by multiple performance measures, are the key criteria.
The nonprofit organization New Leaders is dedicated to promoting student achievement by developing outstanding school leaders to serve in urban schools. New Leaders developed a program to recruit, train, and support school leaders. Researchers from RAND Education, a unit of the RAND Corporation, have been evaluating the New Leaders program over the past five years and measuring the program's impact on student achievement. Because New Leaders has developed principals to serve in multiple states and in both charter and traditional public schools over more than a decade, the researchers have confronted and addressed many evaluation challenges. The lessons learned will be useful for other evaluators of efforts to improve school leadership and for policymakers deciding how to use student outcomes to measure an effort's impact. The challenges and recommendations highlighted here focus on evaluations of improvement efforts rather than evaluations of individual principals, although some may be relevant to both.
Improvement efforts that target school leadership typically affect student outcomes through the principal's influence on teachers. This indirect relationship between principals and students may cause a delay between the start of the intervention and sizable changes in student outcomes. Stakeholders will want to consider other outcomes that are available throughout the effort's lifecycle, such as changes in teacher practices, even when an effort is comprehensive enough to support a rigorous outcomes-based evaluation.
The development of metrics for assessing the success of school leadership improvement efforts poses important challenges related to the following aspects of measurement:
The use of multiple qualitative and quantitative measures can help to overcome the challenges that evaluators and policymakers face by providing a more well-rounded evaluation of the school leadership effort's effect. The choice of measures should be guided by the specific pathway through which a policy, practice, or program seeks to influence school leadership and student outcomes.
Finally, both evaluators and policymakers can contribute to careful planning at the beginning of an evaluation and careful interpretation of findings as the evaluation progresses.
As much as possible, evaluators should weigh carefully the decision of which outcome measures and results to emphasize over others. For example, evaluators should use caution with district-created variables and be sure they understand how these are created, especially if the effort covers multiple districts or charter school management organizations. If possible, evaluators should obtain detailed entry and withdrawal codes from the district so that they can differentiate among student mobility, graduation, and dropout. The evaluator can then construct variables that reflect these different exit pathways.
Evaluators should consider the size and structure of a school's leadership team and the degree of influence the principal has over that team when evaluating efforts to improve principal effectiveness. Because districts provide differing levels of principal autonomy over staffing, this influence is likely to vary by district.
Policymakers have a role to play in mitigating the challenges that evaluators face. They should lay the groundwork for the collection and retention of data that will be needed for the purposes of evaluation at the start of an initiative. There also may be a role for state policymakers to encourage consistent and systematic reporting of core variables of interest by all districts. And developing a data system that allows for more rapid updating of district data would allow the inclusion of startup schools in evaluations and would better reflect changing school composition.
Policymakers are strongly cautioned that, when assessing the effects of efforts targeting principals, it is important to understand and question the methods used to create a comparison group for evaluation purposes and to consider that there may be underlying reasons why one group may differ vastly from another.
As context and grouping of schools matter, policymakers should assess the extent to which the results of other studies are generalizable to their school, district, or state. In adopting an improvement effort that has been effective in another context, policymakers should pay particular attention to whether the new context has similar features and consider the potential implications of any differences.