Incorporating Child Assessments into State Early Childhood Quality Improvement Initiatives

Quality rating and improvement systems (QRISs) have become an increasingly popular mechanism in states and localities seeking to raise the quality of their early care and education (ECE) programs. Typically, QRISs provide ratings of ECE programs based on multicomponent assessments designed to make quality transparent to parents and funders, as well as feedback, technical assistance, and financial incentives tied to ratings to both motivate and support quality improvement (QI).

The ultimate objective of these and other QI initiatives is to ensure that ECE programs promote child development, yet QRISs rarely directly assess children's functioning as a way to evaluate whether specific programs or the ECE system as a whole are improving child outcomes. This stems from the challenges of reliably measuring child functioning and quantifying the contribution of any given ECE program to a child's developmental progress. Thus, QRISs tend to focus on the quality of the inputs in ECE programs rather than on the resulting child developmental outcomes.

In support of state QI initiatives, a RAND study set out to confront these challenges and identify options for incorporating child assessments into the design, implementation, and evaluation of QRISs and other QI initiatives. Drawing on prior research and states' experiences, the study appraises the merits of alternative approaches and offers guidance to designers and policymakers seeking to improve ECE quality.

Recognizing the Challenges of Assessing Young Children

Whereas K–12 accountability systems typically rely on measures of student performance, QRISs focus on the inputs into caregiving and caregiving processes, given the challenges of assessing the developmental progress of children before the age when they can be evaluated by standardized tests. These challenges include obtaining reliable and valid measurement of child functioning. Even with children close to entering kindergarten, attention spans are limited, skills are unevenly developed, and there is discomfort with strangers and strange situations.

Moreover, most of the available tools for assessing young children's development were designed for low-stakes purposes, such as research studies and child performance monitoring in group settings, rather than the increasingly high stakes attached to QRISs with their public ratings, tiered reimbursement systems, and conditional technical assistance. The available tools can be formal or informal, ranging from standardized assessments to home inventories, portfolios, running records, and observation in the course of children's regular activities. The RAND study focused on two of the three main purposes of assessments (using screening of individual children to identify possible handicapping conditions was not a focus of the study as this use is not a program-related issue):

• Supporting and improving teaching and learning. These tools may be formal or informal ways to evaluate how children are developing in multiple domains, so that adult-child interactions, curricula, and other interventions can be tailored to meet an individual child's needs.
• Evaluating of interventions. These tools are designed to meet rigorous research standards including reliability (i.e., consistent measurement) and validity (i.e., quality indicators measure quality well in a given context). Assessors must be carefully trained to meet these standards.

Ultimately, the choice of which assessment tool to use and how to conduct the measurement should be guided by the purpose of the assessment and the way the resulting data will be used.

Framing Approaches to Incorporating Child Assessments

In light of the challenges of assessing young children, the study defined five strategies for using assessments of child functioning to improve ECE quality (see the table), either by improving the inputs to care by tracking children's progress toward key developmental milestones (Approaches A and B) or by employing measures of child development as outcomes to determine the effectiveness of a specific program, the QRIS, or the ECE system as a whole (Approaches C, D, and E).

• With Approach A, ECE caregivers or teachers are trained and expected to use child assessments to evaluate children in their care as a mechanism for improving their caregiving and instructional practice. Aggregated results from the assessments may also be used by program leaders to clarify needs for professional development and program improvement. This approach, which is featured in ECE program accreditation standards and standards for postsecondary ECE teacher preparation programs, does not require a QRIS.
• Approach B has the same objective as Approach A, but it has an explicit link to a QRIS because the demonstrated use of child assessments to support teaching and learning is a required element in the rating system. This approach has already been adopted in a number of state and local QRISs where the use of assessments is a required element, usually to reach the upper tiers of their rating systems.
• Under Approach C, child developmental assessments, collected by specially trained assessors, are used to measure the effect of a specific program on the developmental progress of the participating children, and the results are incorporated into a program's rating. Although this approach is appealing in capturing the outcome of interest in the QRIS—whether an ECE program is improving child functioning—there are significant methodological concerns and it can be costly to implement in a valid and reliable way. This approach is similar to the use of value-added modeling in K–12 accountability systems, a method that remains controversial because of numerous methodological issues. As these concerns are compounded in the early childhood context, this approach has not been used to date in a QRIS.
• With Approach D, child assessments are employed to validate the rating portion of the QRIS. As with Approach C, independent assessors measure children's developmental progress over time to determine if programs with higher QRIS ratings (largely based on program inputs) produce larger developmental gains for participating children (the program outputs). Although the required methods are complex, the approach can be implemented with a sample of children, so it is less costly than Approach C. Such a validation study has been conducted for two state QRISs to date, and similar evaluations are planned in several other states.
• Under Approach E, child assessments and appropriate rigorous methods are used to evaluate the causal effect of specific ECE programs or the ECE system as a whole. Examples of this approach include the quasi-experimental evaluations of a number of state prekindergarten programs. These evaluations have not been connected to a state QRIS, but such an evaluation could be a required one-time or periodic part of a QRIS to determine if the ECE system (or a specific ECE program) is achieving its goal of promoting child development.

Appraising the Merits of Alternative Approaches

Based on a review of each of the five approaches in terms of their overall strengths and weaknesses and the potential benefit relative to cost, the study offers the following guidance to QRIS designers and policymakers seeking to improve ECE program quality (see the table):

• Promote the use of child assessments by ECE caregivers and teachers to improve practice either as part of a QRIS (Approach B) or through other mechanisms (Approach A). This approach, long recognized by ECE professional bodies as good practice, has the potential to support effective individualized care and learning and guide quality improvement efforts at the program level. It does require the needed professional development supports to ensure that assessments are used effectively by caregivers and teachers.
• Implement a QRIS validation study (Approach D) when piloting the implementation of a QRIS and periodically once the QRIS is implemented at scale. Validating the rating portion of a QRIS can be a cost-effective part of a QI system. It can be especially valuable during a QRIS pilot phase when there is an opportunity to identify weaknesses in the rating system design and make modifications before full-scale implementation.
• Implement a statewide evaluation of specific ECE programs or the broader ECE system (Approach E) on a periodic basis regardless of whether a QRIS exists. This approach, especially when used periodically to evaluate publicly funded ECE programs, fulfills a need for accountability and can contribute to quality improvement.
• Proceed with caution if considering a QRIS rating component that is based on estimates of a program's effect on child developmental outcomes (Approach C). Although the motivation for this approach has merit, the available methods are not sufficiently well developed to justify large-scale implementation in high-stakes contexts. Jurisdictions considering this approach should begin with pilot efforts to assess feasibility, cost, and return on investment.

In sum, QRIS designers have several feasible and complementary approaches available for getting to the outcome of interest: child cognitive, social, emotional, and physical functioning. Knowing the merits and drawbacks of each approach, policymakers can determine the optimal mix of strategies given the goals for their ECE system, the available resources, and the anticipated returns.

Topics

• Early Childhood Education
• Education Policy
• Educational Program Evaluation
• Infants
• Preschool Children
• Toddlers