The Validity of Gains in Scores on the Kentucky Instructional Results Information System (KIRIS)

by Daniel Koretz, Sheila Barron

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 9 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Purchase

Purchase Print Copy

 FormatList Price Price
Add to Cart Paperback154 pages $15.00 $12.00 20% Web Discount

In recent years, in an effort to avoid the degradation of instruction and inflation of test scores that often occurred when educators were held accountable for scores on multiple-choice tests, policymakers have experimented with accountability systems based on performance assessments. The Kentucky Instructional Results Information System (KIRIS), which rewarded or sanctioned schools largely on the basis of changes in scores on a complex, partially performance-based assessment, was an archetype of this wave of reform. It is not a given, however, that performance assessment can avoid the inflation of scores that arises when teachers and students focus too narrowly on the content of the assessment used for accountability rather than focusing on the broad domains of achievement the assessment is intended to measure. Accordingly, this study evaluated the extent to which the large performance gains shown on KIRIS represented real improvements in student learning rather than inflation of scores. External evidence of validity — that is, comparisons to other test data — suggests that KIRIS gains were substantially inflated. Even though KIRIS was designed partially to reflect the frameworks of the National Assessment of Educational Progress (NAEP), large KIRIS gains in fourth-grade reading from 1992 to 1994 had no echo in NAEP scores. Large KIRIS gains in mathematics from 1992 to 1994 in the fourth and eighth grades did have some echo in NAEP scores, but Kentucky's NAEP gains were roughly one-fourth as large as the KIRIS gains and were typical of gains shown in other states. The large gains high-school students showed on KIRIS in mathematics and reading were not reflected in their scores on the American College Testing (ACT) college-admissions tests. KIRIS science gains were accompanied by ACT gains only one-fifth as large. Internal evidence of validity — that is, evidence based on patterns within the KIRIS data themselves — was more ambiguous but also provided some warning of likely inflation, particularly in mathematics. For example, schools that showed large gains on KIRIS also tended to show larger than average discrepancies in performance between new and reused test items, suggesting that teachers had coached students narrowly on the content of previous tests. The findings of this study indicate that inflation of scores remains a risk in assessment-based accountability systems even when they rely on test formats other than multiple choice. There is a clear need to evaluate the results and effects of assessment-based accountability systems, and better methods for evaluating the validity of gains need to be developed.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Description of KIRIS

  • Chapter Three

    A Framework for Evaluating the Validity of Score Gains

  • Chapter Four

    Trends in KIRIS Scores

  • Chapter Five

    Past Reports Pertaining to the Validity of KIRIS Gains

  • Chapter Six

    Evidence Pertaining to Retention in Grade

  • Chapter Seven

    External Evidence from NAEP 36

  • Chapter Eight

    External Evidence from the ACT

  • Chapter Nine

    Internal Evidence from KIRIS Data Through 1995

  • Chapter Ten

    Characteristics of KIRIS and Nongeneralizable Gains

  • Chapter Eleven

    Discussion and Implications

This report is part of the RAND Corporation Monograph report series. The monograph/report was a product of the RAND Corporation from 1993 to 2003. RAND monograph/reports presented major research findings that addressed the challenges facing the public and private sectors. They included executive summaries, technical documentation, and synthesis pieces.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.