Given the focus on accountability in education, stakeholders are interested in evaluating whether efforts aimed at improving school leadership show results; the key criteria are student outcomes. This report describes challenges that states, districts, and other entities can expect to encounter as they evaluate efforts targeting school leadership as a way to improve student outcomes and offers suggestions for dealing with those challenges.

Addressing Challenges in Evaluating School Principal Improvement Efforts

Susan Burkhauser, Ashley Pierson, Susan M. Gates, Laura S. Hamilton

ResearchPublished Oct 4, 2012

Cover: Addressing Challenges in Evaluating School Principal Improvement Efforts
Download PDF
Additional Downloads

Includes other ebook formats

State and district policymakers, as well as other organizations, such as foundations and nonprofits, are emphasizing efforts targeting school leadership as a way to improve student outcomes. Given the focus on accountability in education, policymakers and funders are interested in evaluating whether efforts aimed at improving school leadership show results; the key criteria are gains in student achievement. The use of multiple performance measures, including student achievement outcomes, is becoming standard practice in evaluation of efforts targeting both teachers and school leaders. This report describes the challenges that states, districts, and other entities can expect to encounter as they evaluate these efforts and offers suggestions for dealing with those challenges. RAND Education, a unit of the RAND Corporation, is engaged in a multiyear evaluation of the New Leaders program. New Leaders is an organization that recruits, selects, prepares, and supports school leaders to serve in urban schools. Through this project, the researchers have gained practical experience in the issues involved in evaluating efforts that are designed to improve school leadership. The lessons highlighted in this report derive from this experience. The challenges identified in this report can be mitigated through efforts to improve the availability and quality of data, by choosing suitable evaluation methods, and by appropriately interpreting the results of the evaluation.

Key Findings

Challenges in Using Student Outcome Measures Include the Following:

  • Inconsistency in outcome measures
  • Measure manipulation
  • Tracking students across districts
  • Lack of adequate high school outcome measures
  • Effects of student dropout
  • Timing of data and impact

Challenges in Controlling for Student Characteristics Include the Following:

  • Unobserved characteristics
  • Observed characteristics

Challenges in Accounting for School Context Include the Following:

  • Determining appropriate comparison schools
  • Developing measures of school context
  • Measuring principal impact in differing contexts

Challenges in Controlling for Principal Characteristics Include the Following:

  • Quality and availability of principal tenure data
  • Variation in principal career and training paths

Recommendations

  • Multiple measures of principal practice, particularly in relation to organizational management, administrative tasks, and instructional tasks, should be incorporated into the evaluation. Ideally, these multiple measures will include both qualitative and quantitative techniques and consist of both student outcomes and interim measures.
  • Districts (and states, if applicable) should ensure that the same indicators are tracked year to year to the extent possible. When changes are made, they should be clearly documented.
  • Evaluators should determine what measures are being used in personnel evaluations of principals or other administrators when choosing measures to use in the evaluation of an effort to improve school leadership. Evaluators should also consider the likelihood that the evaluation of the overall effort could be biased by measure manipulation by principals.
  • Multiple student outcome measures (such as attendance, dropout and graduation, transfer rate, and college enrollment) should be used in addition to test scores.
  • Evaluators and policymakers should recognize that the time frame for observing a measurable effect on student outcomes from efforts to improve principals may be substantially longer than for efforts that target teachers or students directly. Stakeholders will want to consider other measures that are available earlier in the effort's life cycle.
  • Policymakers should assess the data situation and lay the groundwork for the collection and retention of data that will be needed for the purposes of evaluation at the start of an initiative. State policymakers should encourage consistent and systematic reporting of core variables of interest by all districts.

Topics

Document Details

  • Copyright: RAND Corporation
  • Availability: Web-Only
  • Year: 2012
  • Pages: 39
  • Document Number: OP-392-NLNS

Citation

RAND Style Manual
Burkhauser, Susan, Ashley Pierson, Susan M. Gates, and Laura S. Hamilton, Addressing Challenges in Evaluating School Principal Improvement Efforts, RAND Corporation, OP-392-NLNS, 2012. As of September 19, 2024: https://www.rand.org/pubs/occasional_papers/OP392.html
Chicago Manual of Style
Burkhauser, Susan, Ashley Pierson, Susan M. Gates, and Laura S. Hamilton, Addressing Challenges in Evaluating School Principal Improvement Efforts. Santa Monica, CA: RAND Corporation, 2012. https://www.rand.org/pubs/occasional_papers/OP392.html.
BibTeX RIS

This work was sponsored by New Leaders. The research was conducted in RAND Education, a unit of the RAND Corporation.

This publication is part of the RAND occasional paper series. RAND occasional papers were products of RAND from 2003 to 2013 that included informed perspectives on a timely policy issue, discussions of new research methodologies, essays, papers presented at a conference, and summaries of work in progress.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.