Cover: Addressing Challenges in Evaluating School Principal Improvement Efforts

Addressing Challenges in Evaluating School Principal Improvement Efforts

Published Oct 4, 2012

by Susan Burkhauser, Ashley Pierson, Susan M. Gates, Laura S. Hamilton

Download eBook for Free

FormatFile SizeNotes
PDF file 0.2 MB Best for desktop computers.

Use Adobe Acrobat Reader version 10 or higher for the best experience.

ePub file 1.6 MB Best for mobile devices.

On desktop computers and some mobile devices, you may need to download an eBook reader to view ePub files. Calibre is an example of a free and open source e-book library management application.

mobi file 0.3 MB Best for Kindle 1-3.

On desktop computers and some mobile devices, you may need to download an eBook reader to view mobi files. Amazon Kindle is the most popular reader for mobi files.

Research Question

  1. What challenges can states, districts, and other entities expect to encounter as they evaluate efforts targeting school leadership as a way to improve student outcomes, and how should they deal with those challenges?

State and district policymakers, as well as other organizations, such as foundations and nonprofits, are emphasizing efforts targeting school leadership as a way to improve student outcomes. Given the focus on accountability in education, policymakers and funders are interested in evaluating whether efforts aimed at improving school leadership show results; the key criteria are gains in student achievement. The use of multiple performance measures, including student achievement outcomes, is becoming standard practice in evaluation of efforts targeting both teachers and school leaders. This report describes the challenges that states, districts, and other entities can expect to encounter as they evaluate these efforts and offers suggestions for dealing with those challenges. RAND Education, a unit of the RAND Corporation, is engaged in a multiyear evaluation of the New Leaders program. New Leaders is an organization that recruits, selects, prepares, and supports school leaders to serve in urban schools. Through this project, the researchers have gained practical experience in the issues involved in evaluating efforts that are designed to improve school leadership. The lessons highlighted in this report derive from this experience. The challenges identified in this report can be mitigated through efforts to improve the availability and quality of data, by choosing suitable evaluation methods, and by appropriately interpreting the results of the evaluation.

Key Findings

Challenges in Using Student Outcome Measures Include the Following:

  • Inconsistency in outcome measures
  • Measure manipulation
  • Tracking students across districts
  • Lack of adequate high school outcome measures
  • Effects of student dropout
  • Timing of data and impact

Challenges in Controlling for Student Characteristics Include the Following:

  • Unobserved characteristics
  • Observed characteristics

Challenges in Accounting for School Context Include the Following:

  • Determining appropriate comparison schools
  • Developing measures of school context
  • Measuring principal impact in differing contexts

Challenges in Controlling for Principal Characteristics Include the Following:

  • Quality and availability of principal tenure data
  • Variation in principal career and training paths


  • Multiple measures of principal practice, particularly in relation to organizational management, administrative tasks, and instructional tasks, should be incorporated into the evaluation. Ideally, these multiple measures will include both qualitative and quantitative techniques and consist of both student outcomes and interim measures.
  • Districts (and states, if applicable) should ensure that the same indicators are tracked year to year to the extent possible. When changes are made, they should be clearly documented.
  • Evaluators should determine what measures are being used in personnel evaluations of principals or other administrators when choosing measures to use in the evaluation of an effort to improve school leadership. Evaluators should also consider the likelihood that the evaluation of the overall effort could be biased by measure manipulation by principals.
  • Multiple student outcome measures (such as attendance, dropout and graduation, transfer rate, and college enrollment) should be used in addition to test scores.
  • Evaluators and policymakers should recognize that the time frame for observing a measurable effect on student outcomes from efforts to improve principals may be substantially longer than for efforts that target teachers or students directly. Stakeholders will want to consider other measures that are available earlier in the effort's life cycle.
  • Policymakers should assess the data situation and lay the groundwork for the collection and retention of data that will be needed for the purposes of evaluation at the start of an initiative. State policymakers should encourage consistent and systematic reporting of core variables of interest by all districts.

This work was sponsored by New Leaders. The research was conducted in RAND Education, a unit of the RAND Corporation.

This report is part of the RAND occasional paper series. RAND occasional papers may include an informed perspective on a timely policy issue, a discussion of new research methodologies, essays, a paper presented at a conference, or a summary of work in progress. All RAND occasional papers undergo rigorous peer review to help ensure that they meet high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.