Research Brief

Key Findings

  • The students participating in supplemental educational services generally improved their performance in both reading and math.
  • For students who took advantage of the school choice option and moved to higher-achieving schools, there was no observable achievement gain as a result of changing schools.
  • Overall, those who took advantage of school choice and supplemental services generally came from the disadvantaged populations that NCLB is intended to target.

For decades, parents, educators, and policy-makers have sought ways to raise the quality of education in U.S. schools. The No Child Left Behind Act (NCLB), passed by the U.S. Congress in 2001, was intended to ensure that all children in U.S. public schools are proficient in reading and math by 2014. In addition to creating greater school accountability, NCLB also creates new educational options for parents whose children attend Title I schools that are identified for improvement.[1] Schools are identified for improvement if they have not made adequate yearly progress toward meeting state standards for two or more years.

The first of these options is the opportunity for parents to transfer a child to a school that has not been identified for improvement. This option is available if the child's school is in year 1 or later of identified-for-improvement status. The second option is the opportunity for low-income parents to enroll the child in supplemental educational services—such as tutoring, remediation, or other academic instruction—offered by a state-approved provider, in addition to instruction provided during the school day. This additional option is available if the school is in year 2 or beyond of identified-for-improvement status.

As part of a federally funded study of NCLB's effects, RAND researchers, in collaboration with others, analyzed who chose to participate in each educational option and how each option affected student achievement in nine large, urban school districts across the country.

They found that supplemental educational services had a statistically significant positive effect on student achievement in both reading and math in most districts. They did not find an achievement effect associated with participation in Title I school choice, but the number of participants in most districts was quite small, suggesting the need for caution in interpreting the absence of an observed effect. They also found that users of these two options largely came from the low-achieving or disadvantaged populations that the options are intended to serve.

The Study Analyzed Student Performance Over Time

To estimate the impact of educational options, researchers took into account the fact that students participating in school choice or supplemental educational services were likely to differ from nonparticipants in terms of motivation and other characteristics. The researchers controlled for these "unobservable" characteristics by comparing students to themselves before and after participation. Then, they compared the achievement gains of these students to the gains of all other students. In essence, the researchers used the preparticipation achievement gains of participating students as a control to evaluate the gains of the same students while they were participating in school choice or supplemental services.

Students Participating in Supplemental Educational Services Generally Improved in Reading and Math

In five out of the seven districts in which there were sufficient numbers of students to analyze the effects, the students participating in supplemental educational services showed statistically significant positive effects in both reading and math. Moreover, for those students using supplemental services for multiple years, the analysis suggests that the positive effect might accumulate over time in many of these districts.

School Choice Did Not Produce Observable Achievement Gains

NCLB allows students to transfer out of schools that are failing to meet their state academic standards to schools that are meeting those standards. Although small in number, the students who took advantage of the school choice option generally moved to schools with higher levels of average achievement than the schools from which they came. However, in the six districts in which the number of students was sufficient for statistical analysis, there was no observable achievement gain as a result of changing schools. This was true for African-American students, Hispanic students, and students with disabilities. However, it should be noted that the limited number of students participating in the school choice option may have constrained the ability to detect effects.

Some Groups Took Advantage of Educational Options More Than Others Did

In addition to examining achievement effects, researchers explored participation rates among different groups of students. First, students who participated in Title I supplemental educational services had lower achievement scores than the eligible student population had. Students who participated in Title I school choice had achievement scores that were nearly identical to those of the eligible student population. Also, elementary-grade students participated in school choice and supplemental educational services at rates about five times as high as those for high school students. Among racial and ethnic groups, African-American students had the highest rates of participation in supplemental services. Hispanic students had higher participation rates than white students in supplemental educational services but lower participation rates in school choice. Students with limited English proficiency and those with disabilities had relatively high participation rates in supplemental educational services and relatively low participation rates in school choice. Overall, the users of these options generally came from the disadvantaged populations that NCLB is intended to target.

Conclusions

Title I supplemental educational services showed some promise as a means of raising achievement among disadvantaged students and those most affected by underperforming schools. Given that the operational characteristics—e.g., class sizes, teacher qualifications, hours of service, use of technology—of supplemental educational service providers vary widely, further research might help to identify the characteristics of providers that are particularly effective in raising student achievement.

Meanwhile, the effects of NCLB-related school choice remain in question. While this study did not find that school choice had any effect, positive or negative, on student achievement, it might be that participation rates are still too low for effects to be identified.

Notes

  • [1] Title I schools operate programs funded under Title I of the Elementary and Secondary Education Act of 1965 (ESEA). NCLB is the most recent reauthorization of ESEA.

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.