In 2013, the Delaware Office of Early Learning contracted with the RAND Corporation to conduct an independent evaluation of Delaware Stars for Early Success, the state's quality rating and improvement system for early learning and care programs. The research evaluated the extent to which rating tiers reflected differences in the quality of home- and center-based programs. This final report summarizes all findings of the evaluation.

Cover: Evaluation of Delaware Stars for Early Success
Read Online

Evaluation of Delaware Stars for Early Success

Final Report

by Lynn A. Karoly, Heather L. Schwartz, Claude Messan Setodji, Ann C. Haas

Download eBook for Free

FormatFile SizeNotes
PDF file 3.1 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. Do early care and education programs with higher ratings in the QRIS deliver higher-quality care and early learning than those with lower ratings? What is the relationship between program characteristics and quality in Delaware Stars?
  2. Do children in programs with higher ratings in the QRIS have better learning and developmental outcomes than children in programs with lower ratings? What dimensions of Delaware Stars program ratings are most vital to child learning and developmental outcomes?
  3. How well do the Delaware Stars system components, including technical assistance and financial incentives, operate? What do consumers understand about Delaware Stars?

In 2013, the Delaware Office of Early Learning contracted with the RAND Corporation to conduct an independent evaluation of Delaware Stars for Early Success, the state's quality rating and improvement system (QRIS) for early learning and care programs. The project was designed to provide objective and rigorous empirical evidence of the extent to which rating tiers reflected differences in the quality of home- and center-based programs and whether the system was operating effectively in terms of technical assistance, financial support, and other features. This final report summarizes all findings of the evaluation. The primary focus is on two questions: Do early care and education programs with higher ratings in the QRIS deliver higher-quality care and early learning than those with lower ratings? Do children in programs with higher ratings in the QRIS have better learning and developmental outcomes than children in programs with lower ratings? These questions are examined using data collected in 2014–2015 on program quality from a sample of Delaware center- and home-based providers, along with measures of learning for children enrolled in the sampled programs. The authors also report on results from a survey of the directors of the sampled providers. Other components of the evaluation are also addressed in this report, including findings regarding quality improvement supports, financial incentives, and other aspects of system performance, some of which were analyzed in more detail in two earlier reports. Based on the findings, recommendations are offered for improving the effectiveness of the QRIS.

Key Findings

Delaware Stars Ratings and Program Quality

  • Alternative measures of program quality increased modestly as Delaware Stars rating levels rose.
  • For the most part, alternative measures of program quality were not positively related to scores on the components that make up the Delaware Stars ratings.

Delaware Stars Ratings and Child Developmental Outcomes

  • Differences in children's development across rating tiers were generally small and statistically insignificant.
  • Children from low-income families did not experience differential learning in higher-rated programs when compared with lower-rated programs.
  • Two quality domains of Delaware Stars were predictive of modest differences in selected children's outcomes.
  • These findings for Delaware Stars are consistent with those from QRIS validation studies in other states.

Delaware Stars System Performance

  • Delaware Stars makes a substantial investment in technical assistance (TA), and TA is highly valued, although there may be room for improvement.
  • Financial incentives represent another area of significant investment for Delaware Stars, and directors view the funds as valuable for improving quality.
  • Low parental awareness of Delaware Stars limits value to providers from participating in Delaware Stars.
  • Programs face challenges in advancing through the rating tiers, but they are motivated to improve quality.

Recommendations

  • Learn from other QRIS validation studies. With nearly two dozen states involved in QRIS evaluations, the Delaware Office of Early Learning (OEL) should look beyond the findings of any one study for any particular QRIS and discern the broader findings with relevance for QRIS design and implementation.
  • Consider further refinements to the Delaware Stars rating structure. OEL should consider a simplified, streamlined rating system based on measures of the dimensions of quality that appear to matter most for achieving the goals of the QRIS, such as improvements in child developmental outcomes.
  • Strengthen quality improvement supports in Delaware Stars. With respect to TA in particular, there is scope for lowering TA caseloads, increasing TA training, and improving the guidance that TAs provide.
  • Strengthen the marketing of Delaware Stars to families, drawing on best-practice guidance in the field. Any marketing activities and their timing would need to account for any further planned modifications to Delaware Stars.
  • Enhance administrative data systems to support ongoing system monitoring and quality improvement. OEL should invest in a data manager to revise the Delaware Stars database both by reducing the number of data elements collected and by improving the accuracy of the data elements that remain.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Data Sources and Measures

  • Chapter Three

    Delaware Stars Ratings and Program Quality

  • Chapter Four

    Delaware Stars Ratings and Child Developmental Outcomes

  • Chapter Five

    Delaware Stars System Performance

  • Chapter Six

    Key Findings and Recommendations

  • Appendix A

    Additional Documentation for Delaware Stars

  • Appendix B

    Data Collection Procedures and Response Rates

  • Appendix C

    Data Collection Measures and Instruments

  • Appendix D

    Additional Characteristics of the Provider Sample

  • Appendix E

    Statistical Methods

  • Appendix F

    Additional Documentation of Analyses

  • Appendix G

    Sensitivity Analyses

The research described in this report was sponsored by the Delaware Office of Early Learning and conducted jointly in RAND Education and RAND Labor and Population.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.