How Does Personalized Learning Affect Student Achievement?

Photo by PeopleImages/DigitalVision/Getty Images

Interest in schoolwide personalized learning (PL) models is growing across the nation. While PL and its focus on individualized instruction seems to be a promising concept, a critical question remains: Does PL improve student learning more than other educational approaches? RAND Education experts partnered with the Bill and Melinda Gates Foundation to begin to answer this question.

Personalized learning (PL) refers to practices that tailor the pace and focus of instruction to address the needs and goals of each student. In recent years, schools and school districts have begun to adopt schoolwide PL models. Because the PL approach is sensitive to student interests and areas of academic strength or weakness, the programs have the potential to increase student learning and engagement.

While PL is promising in theory, there are very few evaluations of students’ learning outcomes in such programs. The research described in this brief begins to fill this gap by presenting an evaluation of PL schools and student achievement based on fall 2014 and spring 2015 mathematics and reading test scores. The schools received funding from the Next Generation Learning Challenges (NGLC) initiative to support highly personalized approaches to learning. The study, conducted by RAND Corporation researchers for the Bill & Melinda Gates Foundation, compares the achievement of students in PL schools with (1) matched peers attending non-PL schools and (2) national norms. The study also compares achievement effects between charter and noncharter schools implementing PL.

What Is Personalized Learning, and How Does It Differ from Typical Practices?

To date, there is no single definition of PL. The research team distilled the following working definition based on conversations with practitioners and experts in the field:

Personalized learning prioritizes a clear understanding of the needs and goals of each individual student and the tailoring of instruction to address those needs and goals. These needs and goals, and progress toward meeting them, are highly visible and easily accessible to teachers as well as students and their families, are frequently discussed among these parties, and are updated accordingly.

This aspiration contrasts with typical instructional approaches. In many U.S. schools, efforts to meet individual students’ needs may take less priority than efforts to ensure that all students work toward grade-level standards, progress at the same pace as their grade-level peers, or prepare for grade-level tests at the end of the year. In a PL classroom, students’ learning objectives, pace, and content are likely to vary to a greater extent than they would in a non-PL school. The idea behind PL is that personalized instructional approaches and strategies will improve student outcomes in the short term (e.g., stronger rates of growth in achievement) and in the long term (e.g., successful completion of a postsecondary degree or successful transition into a career).

Technology can play a role in supporting the complexity of the personalization process. When properly supported by teachers, it can help students learn independently and work at their own pace. Technology can also enable educators to take a more personalized approach in their teaching efforts and other activities they undertake to support student learning and development.

Photo by Rob/Fotolia

Personalized Learning Makes a Positive Difference in Mathematics Student Achievement

The research team analyzed mathematics and reading scores for approximately 5,500 students in 32 NGLC schools who took the Northwest Evaluation Association (NWEA) Measures of Academic Progress (MAP) mathematics and reading assessments for one academic year: fall 2014 to spring 2015. The team compared the NGLC MAP scores with the scores of students who did not attend NGLC schools, but who were otherwise similar to their NGLC peers in terms of gender, grade levels, starting test scores, and geographic locations.

As Figure 1 shows, the treatment effects were estimated to be approximately 0.09 in mathematics and 0.07 in reading. Only the mathematics estimate is statistically significant. The effect sizes translate to gains of about 3 percentile points; specifically, a student who would have performed at the median in the comparison group is estimated to have performed 3 percentile points above the median in an NGLC school in both math and reading.

Note: Solid bar indicates statistical significance (p < 0.05) after adjustment for multiple hypothesis tests. Percentile gains translate the treatment effect sizes into the amount of improvement experienced by the median student.

Students Started Below National Norms, but Approached Them by the End of the Year

Figure 2 compares PL students’ average performance with that of their grade-level peers across the country. The figure shows that the students in the PL schools started the year significantly below national norms in both mathematics and reading, but moved closer to the norms during the school year. In mathematics, students gained about 2 percentile points but remained significantly below national norms; in reading, students also gained about 2 percentile points and were performing approximately at national norms by spring.

Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests.

Personalized Learning Students Surpassed National Norms After Two Years

There were 16 NGLC schools that had been in operation the prior year, 2013–14, and had administered the MAP assessment in both academic years. To examine growth trajectories in those schools, the research team restricted the sample to students with test scores in fall and spring of both academic years and examined their scores relative to national norms. The results are shown in Figure 3.

In both mathematics and reading, cumulative growth over the two years is evident. Students started significantly below national norms, gained ground after one academic year, and gained further ground the second academic year, placing them above (though not statistically significantly above) national norms at the end of two years. The largest gains on average appeared to occur in the second year. This suggests that PL systems may require some experience before operating at their fullest potential.

For the 16 schools in the study that started implementing PL in 2013 or earlier, restricted to the students present for all four tests.

Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests.

Photo by Rob/Fotolia

Personalized Learning May Benefit Students of All Abilities

The research team also examined the fraction of NGLC students who surpassed comparison students in mathematics and reading achievement during the academic year. The students were divided into five groups based on their starting levels of achievement. Figure 4 shows that across the achievement spectrum, more than half of NGLC students surpassed their comparison students.

These results suggest that PL is benefiting students of all levels of prior achievement. For the lowest four groups, approximately 60 percent of NGLC students surpassed their comparison students in both mathematics and reading; for the highest group, the percentages are in the mid-50s.

Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests.

Charter Schools Showed Bigger Gains Than Schools Operated by Districts

The average positive achievement effects are composed of widely varying estimates for individual schools. A slight majority of schools were estimated to have positive effects, but a few schools were estimated to have large negative effects on student achievement.

Eight NGLC schools in the achievement sample were operated by school districts; the remaining 24 were charter schools. The district schools were all high schools and middle schools, whereas the charter schools included schools serving all grade levels. This analysis calculated the average treatment effects for district and charter schools separately.

As shown in Figure 5, the charter schools performed similarly in both mathematics and reading, with estimated effects near 0.10.

The district schools have smaller estimates in both subjects, and the reading estimate is near zero. Due to the small sample of schools, the district estimates are particularly imprecise. This imprecision, coupled with the differences in the grade levels served by the two groups, indicates that these trends in district versus charter schools should be treated as suggestive but not conclusive.

Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests.

Two Studies: Positive Results That Vary in Different School Contexts

The results for the 2017 study,^[1] discussed above, demonstrated that PL has a slightly positive effect on student learning. Yet, a 2015 study showed larger gains.^[2] Figure 6 compares the results of the two studies.

Why are the results from the two studies so different? Much of the difference is likely due to different study samples. The schools in the 2017 report were mostly secondary schools that were relatively new to implementing PL. The schools included in the 2015 report, on average, had greater experience in implementing PL and were primarily elementary schools. Because of these differences, the results of the two studies are not directly comparable.

Note: Solid bars indicate statistically significant differences from national norms (p < 0.05) after adjustment for multiple hypothesis tests.

Implications

Together, these analyses offer implications for districts and schools interested in or already implementing PL to consider as they move forward.

PL appears to be promising for improving student achievement. The early evidence presented here suggests that PL approaches can help improve achievement for a broad range of students. However, these results must be confirmed using more-rigorous experimental study designs. Moreover, the implementation of PL in schools is still in the early stages of development, and it is not clear what PL practices or combination of practices have the greatest impact.

PL effects differ across contexts. The positive results on average are composed of widely varying estimates for individual schools, including some schools that showed large negative effects. Other publications in this series describe challenges to PL implementation, which may be greater in certain contexts.

The full effects of PL may take some time to emerge. The benefits suggested by two-year analyses suggest that effects are more positive after schools have at least one year of experience implementing PL. It is not yet clear how effects will accumulate over longer durations, as schools and students gain experience with this major change to schooling. As the field matures with greater understanding of effective PL strategies and more-complete packages of curriculum materials and supports, larger and more-consistent positive effects may be possible. Removal of policy barriers that inhibit PL could also contribute to more-complete implementation.

Notes

• [1] John F. Pane, Elizabeth D. Steiner, Matthew D. Baird, Laura S. Hamilton, and Joseph D. Pane, Informing Progress: Insights on Personalized Learning Implementation and Effects, Santa Monica, Calif.: RR-2042-BMGF, 2017 (available at www.rand. org/t/RR2042).
• [2] John F. Pane, Elizabeth D. Steiner, Matthew D. Baird, and Laura S. Hamilton, Continued Progress: Promising Evidence on Personalized Learning, Santa Monica, Calif.: RAND Corporation, RR-1365-BMGF, 2015 (available at www.rand.org/t/RR1365).