Independent Evaluation of California's Race to the Top-Early Learning Challenge Quality Rating and Improvement System

A mid-term report on a study conducted by American Institutes for Research (AIR) and its partners at the RAND Corporation, Survey Research Management, and Allen, Shea & Associates on California's quality rating and improvement system (QRIS) examines the validity of California's QRIS by assessing the extent to which the quality elements measured in the QRIS relate to each other and how well the QRIS ratings align with independent observations of quality. This report provides preliminary findings on the validity of the QRIS ratings: --California's QRIS captures important aspects of quality. --The quality elements in California's QRIS are not redundant; each element measures a distinct aspect of program quality. --Ratings function differently for centers and family child care homes. --Variation in ratings is limited both for centers and for family child care homes. --There is some evidence that the ratings capture meaningful differences in quality: Higher rated programs were found to be of higher quality on some—but not all—independent measures of observed quality. --Calculating ratings by taking an average of all element scores improves the validity of the ratings. Additional study analyses shed light on possible ways to strengthen or simplify the way that ratings are calculated, such as taking an average score across elements to improve concurrent validity results. Although there is some evidence for the validity of California's QRIS ratings, it is still early in the system's implementation to draw firm conclusions. Further, most participating centers at the time of the study were rated at Tiers 3 or 4, and most participating homes were rated at Tiers 2 or 3, which limited our ability to find effects. The findings may differ with a more diverse group of participating programs. The final report in January 2016 also will include a child outcomes study, providing further evidence regarding the validity and reliability of the Race to the Top–Early Learning Challenge (RTT-ELC) QRIS in California.