Assessing the Validity of the Qualistar Early Learning Quality Rating and Improvement System as a Tool for Improving Child-Care Quality
Jun 10, 2008
An assessment of a quality rating and improvement system (QRIS) used to measure quality in child-care centers finds that much work needs to be done before such systems can be confidently designed and implemented at scale. Also, the absence of a strong link between QRIS ratings and improved child outcomes raises the question of whether research and policy should focus on such outcomes or instead emphasize program outputs.
One increasingly popular approach to improving child-care quality involves developing and implementing quality rating and improvement systems (QRISs), which produce a summary rating to inform parents and provide detailed assessments, hands-on technical assistance, and quality improvement resources to rated child-care providers.
However, little is known about how well QRISs measure or improve quality. With the growing number of proposals (some already implemented) to reward higher-quality child-care programs with higher per-child subsidies, it is critical to validate QRISs. This research brief summarizes a RAND Corporation study of Qualistar Early Learning's QRIS (here-after referred to as the Q-QRIS). Qualistar, a Colorado-based nonprofit, was one of the first organizations to create a QRIS, which Qualistar implemented in 1999. The Q-QRIS includes five components that are generally agreed to contribute to high-quality care: classroom environment, child-staff ratios, staff and director training and education, parent involvement, and accreditation. Points on each component are converted to a summary rating of 0–4 stars.
To assess the Q-QRIS, RAND researchers examined 65 child-care centers and 38 family child-care providers using the Q-QRIS and other measures of quality. Researchers collected data on over 1,300 children in the first wave of data collection and administered the same instruments over two additional waves approximately 12 months apart.
First, the RAND team wanted to understand how well the Q-QRIS components measure quality. Analyses identified significant measurement issues with two components—child-staff ratios and parent involvement. Researchers devoted significant effort to improving them. In particular, parent involvement was poorly measured at the outset; a new Family Partnership measure, based on the idea of a partnership to promote each child's development, was developed and adopted. The new measure shows promise: It elicited some response variation across programs, and it correlated with some other quality measures.
Overall, the research team found that teacher training and education measures still need a good deal of attention. For example, it is unclear how data from multiple teachers should be combined, or how rating systems should deal with teacher movement (if teachers do not remain in a given classroom for very long, it is difficult to assess the impact of their background on process or outcomes). Finally, the study found limited relationships between accreditation status and other measures of quality. This raises the question of whether the cost and effort required to earn national accreditation are justified.
Second, the research team wanted to know how well Q-QRIS component measures correlate with each other. Since all components assess child-care quality, there should be some relationships. However, since each component purportedly measures a different aspect of quality, they should not relate too closely. The study found that the component measures correlated moderately well.
Third, the study examined the relationships between the Q-QRIS star ratings, the individual Q-QRIS component measures that yield those ratings, and two commonly used measures of adult-child interaction. (Adult-child interaction is generally agreed to be the most important aspect of child-care quality.) The study found that the star ratings and the Q-QRIS components are generally unrelated to measures of staff-child interaction, but the lack of results may reflect the fact that these latter quality measures were collected in only one classroom per provider.
Fourth, according to the logic model underlying QRISs, an improved child-care environment, characterized by more-responsive caregiving and enriched content, will lead to better outcomes for children. The researchers examined relationships between the star ratings, Q-QRIS components, and child cognitive and social outcome measures. The researchers found few relationships between individual Q-QRIS components and child outcomes and virtually none between star ratings and child outcomes.
Fifth, the researchers examined two subgroups of children—those who came from low-income homes and those who had experienced high doses of child-care exposure—to determine whether in these subgroups there might be relationships between the Q-QRIS components and improved child outcomes that were not apparent in the general study population. The pattern of results for these children did not differ.
Finally, the RAND team examined whether child-care quality improved over time. The researchers found that provider quality did improve, but they could not unequivocally attribute improvement to the Q-QRIS. Improvements may have been a reaction to simply being assessed or were part of regular practice in a group of self-selected providers. The lack of a comparison group and limited implementation data made testing the impact of the intervention impossible.
Together, these findings provide mixed support for the Q-QRIS and its components as measures of provider quality. The Q-QRIS and the component measures correlate moderately with each other and show some relationships with one of the two measures of child-adult interaction that were used as benchmarks. However, the research team found little evidence that the Q-QRIS ratings predict child outcomes.
Definitive conclusions about the validity of the Q-QRIS and its components cannot be drawn because of study design and implementation limitations—including criterion measures collected from a single classroom in each center, data primarily drawn from low-stakes settings, a new measure of parental involvement that has yet to be validated, lack of a randomized design, nonrandom provider attrition, and very high child attrition. These limitations also make it difficult to generalize study findings to other QRISs.
Study findings indicate that building a QRIS takes time and should probably be done incrementally. Each construct should be clearly articulated, designed, tested, and validated in the context in which it will be used. Once the components are well measured, an iterative, evidence-based validation of the QRIS as a whole can begin. A focus on measurement research will slow the rollout of quality rating and improvement systems but should produce better systems. These findings have led the study authors to work with other stakeholders to develop a QRIS consortium that will devote resources to sharing data and conducting the many research studies to provide an empirical basis for QRISs.
The study did not find strong associations between the Q-QRIS and child outcomes—findings that are consistent with other studies in the field. This lack of relationships raises the broader question of which, if any, child outcomes should be promised. For example, early childhood educators, researchers, and kindergarten teachers are more interested in children's capacity to regulate emotions, develop trusting relationships with adults, and approach learning in an efficacious way than in children's pre-academic skills.
Alternatively, until we can build a stronger empirical basis for quality measures, it may be appropriate to ignore longer-term child outcomes entirely, focusing instead on program outputs, such as children's engagement in developmentally appropriate tasks in a safe and supportive environment. Clearly, more research should be directed to this area.