Evidence Mixed on Demand and Impact of Out-of-School Programs
March 3, 2005
Evidence is mixed on whether there is a shortage of publicly funded, group-based programs for children outside of school and on whether such programs improve academic achievement, according to a RAND Corporation study issued today.
The RAND report says few out-of-school programs—those providing services before and after school and on days school is not in session—have been rigorously evaluated. The study says existing evaluations with rigorous methodology show at most that some programs have produced modest positive effects on educational expectations, high school graduation rates, credits earned, college attendance and social behaviors.
The study notes that the past 20 years have seen unprecedented growth in the provision and amount of public financing of out-of-school programs. It concludes that policymakers and program implementers should remain skeptical of claims about pent-up demand for group-based programs as well as claims that these programs can meet multiple needs and impact positively on multiple outcomes. It goes on to say that rapid growth should make way for concentrating on how to improve the quality of offerings by existing programs and of systems of provision.
“Making Out-of-School-Time Matter: Evidence for an Action Agenda” was produced by RAND Education and RAND Labor and Population. The study was commissioned by The Wallace Foundation as part of its work with out-of-school learning and its mission to support and share effective practices. The study also seeks to provide an objective and systematic examination of the out-of-school time literature, and inform the key issues in the ongoing discussions related to how to improve out-of-school time programming.
The RAND report found that previous studies pointing to strong unmet demand for out-of-school programs were based on assumptions of demand that could not be verified, or on opinion polls that did not pose important resource tradeoff questions. It also found that studies of existing programs revealed a significant number of open slots and dropouts, implying that the unmet demand for existing programs is not insistent or pent-up.
“Before expanding publicly funded out-of-school programs, policymakers may want to do further evaluations and work to improve the existing programs so they attract more students and produce the desired results,” said Susan Bodilly, lead author of the study and associate director of RAND Education.
“This study points to a number of ways out-of-school time programs can increase their value to our children,” said M. Christine DeVita, president of The Wallace Foundation. “It calls attention to the importance of developing high-quality programs that attract and retain young people so that they participate long enough to receive the benefits the programs offer. It also brings into sharp relief the relatively poor quality of the program evaluations that have been done to date, most of which failed to reach RAND’s rigorous standards. We look forward to continuing the discussion with researchers, practitioners, policymakers and families about how best to deliver vibrant learning and enrichment opportunities for all our nation’s young people.”
About 17 percent of children ages 5 to 14 with mothers who hold jobs are in out-of-school programs. About 11 percent of children of mothers who are not employed attend some sort of formal program during non-school hours.
Bodilly and Megan Beckett, the other author of the RAND report, offered recommendations for basic information requirements needed to improve out-of-school programming and make the debate more productive. Their study calls for developing:
- Local assessments to clarify demand for specific out-of-school services.
- Rigorous program evaluations, especially for large publicly funded programs.
- Dissemination of standardized ways to measure participation.
- Cost-effectiveness evaluations to compare out-of-school programs to other alternatives.
- Development and dissemination of tools to collect and report cost information.
- Practical ways to improve recruitment and enrollment practices.
- Collection and analysis of data that can be used when making decisions about services and the use of assessment and accountability mechanisms based on established guidelines.
The authors reviewed school-age care literature, youth development literature, and the education literature. They suggested a set of factors as the field’s “best guess about program qualities that might lead to positive outcomes” that could provide general guidance for the design of new programs and improvement of existing programs.
The factors include: a clear mission; high expectations for children; positive social norms; a safe and healthy environment; a supportive emotional climate; a small total enrollment; stable, trained personnel; appropriate content and pedagogy relative to children’s needs and the program’s mission, with opportunities to engage; integrated family and community partners; and frequent assessment.
A printed copy of “Making Out-of-School-Time Matter: Evidence for an Action Agenda” (ISBN: 0-8330-3734-X) can be ordered from RAND’s Distribution Services (email@example.com or call toll-free 877-584-8642).
About the Wallace Foundation
The Wallace Foundation is an independent, national foundation dedicated to supporting and sharing effective ideas and practices that expand learning and enrichment opportunities for all people. Its three current objectives are: strengthening education leadership to improve student achievement; improving out-of-school learning opportunities; and expanding participation in arts and culture. More information and research on these and other related topics can be found at www.wallacefoundation.org or by calling 212-251-9783.