- What challenges can states, districts, and other entities expect to encounter as they evaluate efforts targeting school leadership as a way to improve student outcomes, and how should they deal with those challenges?
State and district policymakers, as well as other organizations, such as foundations and nonprofits, are emphasizing efforts targeting school leadership as a way to improve student outcomes. Given the focus on accountability in education, policymakers and funders are interested in evaluating whether efforts aimed at improving school leadership show results; the key criteria are gains in student achievement. The use of multiple performance measures, including student achievement outcomes, is becoming standard practice in evaluation of efforts targeting both teachers and school leaders. This report describes the challenges that states, districts, and other entities can expect to encounter as they evaluate these efforts and offers suggestions for dealing with those challenges. RAND Education, a unit of the RAND Corporation, is engaged in a multiyear evaluation of the New Leaders program. New Leaders is an organization that recruits, selects, prepares, and supports school leaders to serve in urban schools. Through this project, the researchers have gained practical experience in the issues involved in evaluating efforts that are designed to improve school leadership. The lessons highlighted in this report derive from this experience. The challenges identified in this report can be mitigated through efforts to improve the availability and quality of data, by choosing suitable evaluation methods, and by appropriately interpreting the results of the evaluation.
Challenges in Using Student Outcome Measures Include the Following:
- Inconsistency in outcome measures
- Measure manipulation
- Tracking students across districts
- Lack of adequate high school outcome measures
- Effects of student dropout
- Timing of data and impact
Challenges in Controlling for Student Characteristics Include the Following:
- Unobserved characteristics
- Observed characteristics
Challenges in Accounting for School Context Include the Following:
- Determining appropriate comparison schools
- Developing measures of school context
- Measuring principal impact in differing contexts
Challenges in Controlling for Principal Characteristics Include the Following:
- Quality and availability of principal tenure data
- Variation in principal career and training paths
- Multiple measures of principal practice, particularly in relation to organizational management, administrative tasks, and instructional tasks, should be incorporated into the evaluation. Ideally, these multiple measures will include both qualitative and quantitative techniques and consist of both student outcomes and interim measures.
- Districts (and states, if applicable) should ensure that the same indicators are tracked year to year to the extent possible. When changes are made, they should be clearly documented.
- Evaluators should determine what measures are being used in personnel evaluations of principals or other administrators when choosing measures to use in the evaluation of an effort to improve school leadership. Evaluators should also consider the likelihood that the evaluation of the overall effort could be biased by measure manipulation by principals.
- Multiple student outcome measures (such as attendance, dropout and graduation, transfer rate, and college enrollment) should be used in addition to test scores.
- Evaluators and policymakers should recognize that the time frame for observing a measurable effect on student outcomes from efforts to improve principals may be substantially longer than for efforts that target teachers or students directly. Stakeholders will want to consider other measures that are available earlier in the effort's life cycle.
- Policymakers should assess the data situation and lay the groundwork for the collection and retention of data that will be needed for the purposes of evaluation at the start of an initiative. State policymakers should encourage consistent and systematic reporting of core variables of interest by all districts.
This work was sponsored by New Leaders. The research was conducted in RAND Education, a unit of the RAND Corporation.
This report is part of the RAND Corporation occasional paper series. RAND occasional papers may include an informed perspective on a timely policy issue, a discussion of new research methodologies, essays, a paper presented at a conference, or a summary of work in progress. All RAND occasional papers undergo rigorous peer review to help ensure that they meet high standards for research quality and objectivity.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.