Text-Based Accountability Systems: Lessons of Kentucky's Experiment
Jan 1, 1999
|PDF file||9 MB||
Use Adobe Acrobat Reader version 10 or higher for the best experience.
|Add to Cart||Paperback154 pages||$15.00||$12.00 20% Web Discount|
In recent years, in an effort to avoid the degradation of instruction and inflation of test scores that often occurred when educators were held accountable for scores on multiple-choice tests, policymakers have experimented with accountability systems based on performance assessments. The Kentucky Instructional Results Information System (KIRIS), which rewarded or sanctioned schools largely on the basis of changes in scores on a complex, partially performance-based assessment, was an archetype of this wave of reform. It is not a given, however, that performance assessment can avoid the inflation of scores that arises when teachers and students focus too narrowly on the content of the assessment used for accountability rather than focusing on the broad domains of achievement the assessment is intended to measure. Accordingly, this study evaluated the extent to which the large performance gains shown on KIRIS represented real improvements in student learning rather than inflation of scores. External evidence of validity — that is, comparisons to other test data — suggests that KIRIS gains were substantially inflated. Even though KIRIS was designed partially to reflect the frameworks of the National Assessment of Educational Progress (NAEP), large KIRIS gains in fourth-grade reading from 1992 to 1994 had no echo in NAEP scores. Large KIRIS gains in mathematics from 1992 to 1994 in the fourth and eighth grades did have some echo in NAEP scores, but Kentucky's NAEP gains were roughly one-fourth as large as the KIRIS gains and were typical of gains shown in other states. The large gains high-school students showed on KIRIS in mathematics and reading were not reflected in their scores on the American College Testing (ACT) college-admissions tests. KIRIS science gains were accompanied by ACT gains only one-fifth as large. Internal evidence of validity — that is, evidence based on patterns within the KIRIS data themselves — was more ambiguous but also provided some warning of likely inflation, particularly in mathematics. For example, schools that showed large gains on KIRIS also tended to show larger than average discrepancies in performance between new and reused test items, suggesting that teachers had coached students narrowly on the content of previous tests. The findings of this study indicate that inflation of scores remains a risk in assessment-based accountability systems even when they rely on test formats other than multiple choice. There is a clear need to evaluate the results and effects of assessment-based accountability systems, and better methods for evaluating the validity of gains need to be developed.
Description of KIRIS
A Framework for Evaluating the Validity of Score Gains
Trends in KIRIS Scores
Past Reports Pertaining to the Validity of KIRIS Gains
Evidence Pertaining to Retention in Grade
External Evidence from NAEP 36
External Evidence from the ACT
Internal Evidence from KIRIS Data Through 1995
Characteristics of KIRIS and Nongeneralizable Gains
Discussion and Implications
This report is part of the RAND Corporation Monograph report series. The monograph/report was a product of the RAND Corporation from 1993 to 2003. RAND monograph/reports presented major research findings that addressed the challenges facing the public and private sectors. They included executive summaries, technical documentation, and synthesis pieces.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.