RAND researchers used two types of analyses to identify outcomes: a randomized controlled trial, or causal analysis, in which students were randomly assigned to either a treatment or control group; and correlational analysis that compared specific groups of students, such as high attenders, with the control group. Researchers also collected extensive data on program implementation and attendance — factors that influenced outcomes.
Findings from this study suggest that high attenders benefited from the programs in mathematics and language arts achievement. However, it was hard for some districts to get students to attend regularly.
What happens when we compare the control group with those kids who were randomly selected to participate?
The causal analysis from the randomized controlled trial examined program effects on all treatment students accepted into the program. Because many students did not show up or had low attendance, these effects are smaller than the effects experienced by students who attended regularly.
Students Demonstrated a Near-Term Benefit in Mathematics After the First Summer But No Statistically Significant Benefits from Two Summers of Programming
Causal results from the first summer programs showed a modest boost in mathematics performance, representing about 15 percent of a typical annual gain in mathematics, which had dissipated a year later. There were no significant benefits for any other academic, social-emotional, or behavioral outcomes.
Causal analyses of providing two summers of programming found no statistically significant effects for any of the measures — a result that is influenced by the fact that nearly half the students who were accepted into the programs did not participate in the second summer, effectively diluting the effect on the students who did attend. The higher the no-show rate, the larger the effect of the program would have to be (on those who did attend) in order to be detected.
What happens when we compare the control group with those kids who attended?
Correlational analysis, which compares subsets of the treatment group to the control group, can yield important insights about outcomes and the factors that influence them. It also introduces the possibility of selection bias because differences between subgroups of students (such as high attenders) and the control group could explain differences in outcomes. To help mitigate this risk, researchers controlled for the same broad set of student characteristics as in the causal analyses, including student characteristics and prior academic performance. For this reason, they think the sum of evidence makes it likely that the academic results are due to participation in the summer learning programs.
High Attenders in the First Summer Benefited in Mathematics
High attenders (those who attended at least 20 days) in the first summer gained a significant and substantial benefit in mathematics from the program. We observed this benefit on the spring 2014 state assessments as well, when the high attending treatment students outperformed the control students at a rate representing about
15 percent of an average annual gain in mathematics.
Findings from this study suggest that high attenders benefited from the programs in mathematics and language arts achievement.
High Attenders in the Second Summer Had Better Outcomes in Math and Language Arts
Students who attended at high rates in summer 2014 outperformed control group students in both math and language arts both in the fall and the following spring. They were also assessed as having stronger social and emotional competencies than the control group students.
The academic boost for these students translates to between 20 and 25 percent of typical annual gains in mathematics and language arts.
Because most students who were high attenders in 2014 were also high attenders in 2013, it is not clear whether these positive outcomes are the result of having attended in both summers, or program improvements in the second summer — the research team suspects both.
Students Who Received High Levels of Academic Time on Task Had Better Outcomes in Mathematics and Language Arts
Students who received at least 25 hours of mathematics or 34 hours of language arts instruction (defined as high levels) performed significantly better on the language arts and mathematics tests in fall 2013 and fall 2014 than control group students. After the second summer, these positive effects persisted into the spring.
About 35 percent of attending students received a high number of hours of mathematics and language arts instruction. This calculation considers a student's attendance and the estimated amount of time that their particular classes were focused on academic material, based on researcher observations.
Students Who Received High-Quality Language Arts Instruction Had Better Outcomes in Language Arts
The study found that students who received the highest quality of language arts instruction benefited in that subject in fall 2013 after the first summer. The measure of quality focused on clear instruction, on-task behavior, and teachers' efforts to ensure that all students understood the material. These positive trends emerged in spring and fall of 2014, though they were not statistically significant.
Of Those Selected to Participate, What Percentage Attended?
While about 80 percent of treatment students attended the first summer program in 2013, only about half came in the next summer. To understand the increase in no-shows in 2014, it is important to remember that the numbers represent participation of students who were accepted 14 months earlier to attend the program in two consecutive summers. This time lag increased the possibility that students made other plans for the second summer or moved out of the district.
Of Those Students Who Participated, About 60 Percent Were High Attenders
If we consider only those students who showed up, about 60 percent of them were high attenders in each of the summers.
Taken together, students who came to the program attended about 75 percent of the program days, although these numbers ranged from 60 percent to 80 percent in specific districts. Although districts with the lower average daily attendance rates tried various strategies to improve attendance, their efforts did not have much effect.
Five Tips to Help Guide Summer Program Leaders
Besides offering evidence on effectiveness, both causal and correlational, the study offers guidance to practitioners on how to increase the benefits of summer learning programs of this kind. Three of these lessons highlight the importance of maximizing the amount of instruction students receive over the summer.
Offer programs for at least five weeks. Because longer exposure to instruction is linked to greater benefits, programs should last at least five weeks — and preferably six or more weeks — with at least three hours of academics per day.
Track and maximize attendance rates. Based on study findings on the barriers to consistent attendance, district leaders should consider
offering programs to multiple age levels to reduce the possibility that students will have to stay home to care for younger siblings
creating engaging academic and enrichment opportunities that excite students
employing adults who can focus on student behavior so that bullying and fighting can be minimized
making personal connections with families of students who are less likely to attend regularly: lower-achieving students, lower-income students, and students with poor school-year attendance
establishing mandatory programs for the lowest-performing students, who are less likely to attend the voluntary programs at high rates.
Create schedules that protect instructional time. Ensuring a sufficient amount of academic time on task is critical. Besides trying to maximize attendance, program leaders should protect time for academics by avoiding meal breaks during academic blocks or dismissing students early from academic classes.
Invest in instructional quality. Focusing on instructional quality, particularly for language arts instruction, should benefit students. Program leaders are urged to
recruit from among the district's highly effective teaching pool, prioritizing summer teachers with relevant subject and grade-level experience, who are often better able to connect the summer content to prior or upcoming school-year lessons
encourage teachers to take the time to ensure that each student understands the material
provide teachers with a curriculum that aligns to the school year and state standards, while meeting the needs of all participating students, including low achievers.
Minimize costs by considering probable no-show and attendance rates. Use this study's estimates or district data on no-show and attendance rates when deciding how many teachers to hire, how much space is needed, and other matters that are affected by student numbers.
About RAND's Summer Learning Series
Other reports from this study will appear over the next few years, including an analysis of whether these outcomes persist into the spring of 2017 and a handbook offering guidance to districts that are running summer learning programs.
This report is part of the RAND Corporation Research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.
Citation
Format:
Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan Schweig, Andrew McEachin, and Kyle Siler-Evans, Kids Who Attend More Benefit More: Voluntary Summer Learning Programs. Santa Monica, CA: RAND Corporation, 2016. https://www.rand.org/pubs/research_briefs/RB9924.html.
Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan Schweig, Andrew McEachin, and Kyle Siler-Evans, Kids Who Attend More Benefit More: Voluntary Summer Learning Programs, RAND Corporation, RB-9924-WF, 2016. As of May 29, 2023: https://www.rand.org/pubs/research_briefs/RB9924.html
