Making Out-of-School Time Matter

Programs that offer out-of-school services to children and youth can be found in every state and locale, and they run the gamut from school-aged care services supporting working parents to programs specifically structured to prevent problematic behaviors or to increase academic attainment. A loosely connected set of providers, clients, sponsors, and intermediaries makes up the local markets, referred to as the out-of-school-time (OST) field. This field and the actors in it have been evolving in response to shifts in the economy, the growing demand for services associated with increased numbers of working mothers in the labor force, concerns over youth development or its lack, and increased academic expectations for youth.

In the past 20 years, various groups, including child care practitioners, educators, and juvenile justice experts, have argued that OST programs are part of a solution to problems affecting children and youth. These groups have successfully drawn national attention to OST issues and have worked to increase public funding. But they do not all agree on how to move forward. A heated debate is now taking place over how to increase the positive impact of out-of-school-time programming.

The Research Purpose

As part of its work with out-of-school learning and its mission to support and share effective ideas and practices, the New York City–based Wallace Foundation commissioned RAND to provide an objective and systematic examination of the OST literature to clarify and inform the key issues in the ongoing discussions related to improving OST programming. RAND researchers Susan Bodilly and Megan Beckett undertook a literature review focused on group-based programs that provide care for school-aged children (6 to 18 years of age) during nonschool hours and that also attempt to improve the children's behavioral, social, and academic development outcomes. The authors wanted to capture what is known with some certainty and what is more speculative about claims being made in five major issue areas: (1) the level of unmet demand; (2) the types of outcomes that participation in OST programs is expected to contribute to; (3) determinants of quality in program offerings; (4) determinants of participation; and (5) practices that ensure that quality programming is available to meet local demand.

The results of the study are presented below. The report on that study offers a useful summary and assessment of current knowledge about these OST issues, along with recommendations about what is needed to improve programming and make the current policy debate more productive.

Demand for OST Services

A significant aspect of the current debate concerns whether or not there is unmet demand for programming. Some advocates call on policymakers to provide public funding for universal coverage, and the trend is to push for capacity expansion, seeking to offer more slots to meet the demand for OST programs, such as after-school care, that many claim exceeds availability of existing slots in such programs. The review, however, found that the limited number of studies finding unmet demand for services (1) were based on unverifiable assumptions; (2) often estimated the total possible need for childcare services, rather than what was demanded from providers outside the home; and (3) were based on surveys that did not ask the respondent to consider trade-offs in the use of funds, thereby probably overestimating true demand.

In contrast, studies of existing programs reveal a significant number of open slots and absentees. The review did not find systematic evidence of what it is that parents and youth are demanding, be it a safe environment, improved or better social behaviors, improved academic outcomes, or all of these. It concludes that, while data on current usage are clear, the nature and extent of unmet demand for OST programming are unclear.

Potential Effects of Programs

The review examined evaluations of the outcomes of OST programs. Very few such evaluations have been rigorous—meaning, at a minimum, using a control group against which to compare children who participated in a given OST program. Children who choose to attend OST programs might differ from those who choose not to, in terms of motivation, aspiration, and other factors. Most of the studies reviewed did not control for this self-selection bias, making it difficult to conclude that differences between participants and nonparticipants were wholly attributable to program effects.

Analysis of the most rigorous evaluations suggests that these programs have had, at best, modest positive effects on academic achievement, academic attainment, and reducing risky behaviors such as drug use or teen pregnancy. Documented effects might vary by grade level, background of children, level of participation, program content, site context, and whether the program was well suited to achieve the desired outcome.

These evaluations provide few insights into whether existing programs offer safe and healthy environments. Evaluations of existing programs provide few clues of whether other OST programs would achieve similar academic or behavioral effects. The cost-effectiveness of any of these programs relative to other interventions, including expansion of the school day, is also not well understood.

Program Factors Associated with Positive Outcomes

Studies analyzing which program features are associated with improved outcomes are often not rigorous and tend to rely on expert opinion. The review drew on recent compendia or studies of quality indicators in OST or related settings, including literature on school-aged care, youth development, and effective schools; analyses of class-size-reduction programs; and recent studies on teacher-training effects.

These multiple, albeit less rigorous, studies point to a growing consensus about several program factors that might be associated with improved youth outcomes:

• A clear mission
• High expectations
• A safe and healthy environment
• Supportive emotional climate
• A small total enrollment
• Trained personnel who remain with the program
• Appropriate content and pedagogy relative to the children's needs and the program's mission
• Integrated family and community partners
• Frequent assessment.

Although these factors have not been formally tested in OST programs or tested for effectiveness in rigorous experimental studies, they provide a useful cluster of characteristics upon which to base initial program-improvement efforts. These efforts should be evaluated to determine whether they are, in fact, effective in meeting program goals. The field has moved toward the development and publication of standards for service providers consistent with the above characteristics.

Improving Participation

If quality programming is provided, then it is also worth considering how to improve participation, particularly by children and youth who could most benefit from the services. The review drew on a cross section of fields to understand how to encourage and target participation. It found empirical evidence that participation varies by participant background, implying that targeting services might increase participation. For example, lower-income families might be more attracted to subsidized programs that are conveniently located within their neighborhood.

Practical ways to increase enrollment and attendance in programs have been developed and tested in the job-training and military-recruiting fields. They include identifying possible participants, dedicating sufficient and effective resources for outreach and recruitment, and locating such efforts in places where targeted youth, and their key influencers, congregate. Monitoring attendance and quality, following up on absentees, and offering incentives to programs for achieving high attendance rates are potential ways to improve attendance. Most importantly, successfully reaching targeted groups and providing them appropriate, accessible services require knowledge of the groups' needs and preferences and current services, which might vary from place to place.

Improving Capacity

The review found little rigorous empirical evidence about how to build capacity in the OST field. Studies provided ideas about how to improve and build capacity both of individual programs and across local, regional, and national markets. The ideas included: providers and funders collaborating and planning jointly; collecting and analyzing data on program services to be used to improve programming; issuing standards for the OST field; and providing incentives to providers for offering improved services.

Although these studies did not provide the evidence needed to create a well-crafted agenda for improving community provision of OST services, they pointed the way toward useful demonstrations to test policy ideas.

Implications

Policymakers and program implementers should remain skeptical of claims about unmet demand for programs, as well as of claims that programs can meet multiple needs or produce positive impacts on an array of outcomes. The available evidence suggests that improving quality of offerings in existing programs should take precedence over rapid growth in supply. Designing and implementing effective programs will take careful planning and attention—and probably very significant funding.

A public discussion of the goals of OST would benefit from a better accounting of demand, both in qualitative terms (about what children, youth, and parents want in OST programming) and quantitatively (about how many slots are demanded for different goals). Resources would be well spent in assessing local needs and barriers to participation, and in developing programs to meet those needs and remove those barriers. Furthermore, any push toward rapid expansion of slots should be tempered with an assessment of how that expansion might affect the quality of the programs offered.

Recommendations

To improve OST programming and make the current debate more productive, the report recommends the following types of activities:

• Local assessments, using surveys and other field instruments, to clarify demand for specific services by specific classes of clients, and the level and quality of existing providers.
• Development of forums for public consideration of the results of such analyses.
• More-systematic program evaluations, especially for large, publicly funded programs, with proper controls for self-selection and, where possible, for the effect of participation levels on outcomes; documentation of the effect of different program designs or contexts on outcomes; determination of the effects by age group or class of participant; and attention to measuring cost-effectiveness.
• Dissemination of standardized measures of participation levels and intensity that are used in program evaluations, when possible.
• Development and dissemination of tools to collect and report cost and other information necessary to undertake cost-effectiveness evaluations, with the ultimate goal of comparing OST programs to alternatives.
• Development, demonstration, testing, and evaluation of practical and cost-effective means to improve practices for recruiting and enrolling participants for targeted services.
• Development of effective forums and incentives to disseminate existing standards, guidelines, and best practices, and new ones as they evolve or are discovered through research.
• Support for collection and analysis of data for use in decisionmaking, monitoring, and continuing assessment of OST services.