Evaluating After-School Care

More and more children with mothers who work outside of the home are participating in after-school programs, and increased federal and state funding suggests that the number of such programs will continue to grow in the coming years. Funding for the U.S. Department of Education's 21st Century Community Learning Centers alone increased from $1 million in 1997 to $450 million in 2000, with $1.5 billion proposed for 2002.

The impetus for this growth extends beyond increased government funding. For instance, a partnership between the U.S. Department of Education and the Mott Foundation has given rise to the Afterschool Alliance, a coalition of public and private organizations dedicated to raising awareness of the importance of after-school programs and advocating on their behalf. The alliance's goal is that every child should have access to high-quality, affordable after-school care by 2010.

The "high-quality" goal is particularly daunting. The need for after-school services became apparent so quickly that there has been virtually no time to muster support for testing and evaluating the various aspects of the programs being offered. Program managers who are committed to high-quality care have thus been faced with the challenge of attempting to measure their activities against almost nonexistent standards.

It was within this context that a team of RAND researchers sought to establish a set of model after-school program features or practices (see the box) that are supported by scientific studies where possible, or by expert judgment where not. A second objective was to provide the rating scales and data collection instruments required for evaluators to test programs against model practices.

The research team began by systematically assessing the published research literature on after-school care to identify candidate model practices. They were defined as program or process elements shown or upheld by experts in the field to be associated with high-quality after-school programs or positive child outcomes (such as those associated with educational attainment, emotional development, and health). The researchers found only a few scientifically sound empirical studies that demonstrated an association between certain practices and desirable outcomes. Most of the publications in the literature simply summarized recommendations provided by expert panels and individuals. Therefore, the RAND list of model practices must be considered preliminary and subject to change in light of new research.

Because most of the literature was at varying levels of reliability below the "gold standard" of experimental research, the RAND team placed particular emphasis on finding a way to convey the degree of support for each candidate practice. To rate the candidate model practices in this regard, the RAND researchers adapted conventional meta-analytic techniques for application to an empirically weak field.

In conventional meta-analysis, the effect sizes found in different studies are weighted according to the reliability or scientific rigor of each study and then averaged. In the present case, RAND also classified studies by reliability but, instead of using effect sizes, the values to be weighted were the frequencies with which effects were reported (or judged to exist). The final scores were expressed in qualitative terms—that is, whether a model practice candidate had strong, moderate, or limited support in the literature.

The candidate practices were classified by type and rated according to the level of support for each. Those shown in italics in the box were judged to have strong support. All others listed in the box had moderate support. Two other candidates—hiring and retaining experienced staff and keeping the turnover rate low—were judged as having limited support and were dropped from the list of model practices.

As a further aid to evaluators, the RAND research team devised scales to help evaluators determine whether their program's adherence to each model practice would best be described as "excellent," "good," "minimal," or "inadequate." These scales are specific to each practice—for example, for the staff management practice providing attractive compensation, the break points for excellent, good, and minimal are the 75th, 50th, and 25th percentiles of hourly wages paid to child-care workers in the program's county.

The researchers cautioned that in assessing adherence to model practices, program evaluators should be sensitive to trade-offs and constraints arising from a program's objectives or budget. For example, if a provider sees its mission as offering after-school care to all children in the community who need it, low total enrollment might not be an important goal. High enrollment would therefore be a disadvantage that is deliberately tolerated to permit what the provider regards as the greater value of broad community service. For other programs, the principal objective, for example, may be to keep children safely off the streets after school. In such cases, model practices intended to enhance educational achievement may not be affordable or actively pursued. The evaluator should nevertheless assess the program's adherence to all model practices, but the trade-offs and constraints that program managers face should still be identified.

The scales used to determine adherence to model practices, the survey forms for collecting data supporting application of the scales, and the results of a sample application can be found in the RAND report cited at the end of this brief. This study was funded by Stone Soup Child Care Programs, a provider of after-school care; by RAND, through its Child Policy Project; and by the Promising Practices Network.