Jun 1, 2014
Student performance in mathematics remains a source of concern for U.S. educators and policymakers. Although math scores have risen slightly in recent decades, U.S. students still perform poorly on the National Assessment of Educational Progress and in international comparisons with their counterparts from many other countries. In an effort to address this issue, many districts and schools have turned to computer-based tools as a way to boost math performance. These tools allow self-paced instruction and provide students with customized feedback. These features, it is widely held, will improve student engagement and improve proficiency. However, evidence to support these claims remains scarce. In many cases, these tools have been adopted with little or no evaluation.
To make headway in addressing this knowledge gap, a team of RAND researchers assessed whether a popular algebra curriculum that includes tutoring software would be effective in improving the math test scores of middle and high school students. Cognitive Tutor Algebra I (or CTAI), developed by Carnegie Learning, is a first-year algebra course that blends classroom instruction and textbook-based activities with computer-based instruction and has shown efficacy in improving math performance in isolated, small-scale demonstrations. Algebra is of particular interest because it can function as a gateway subject that leads students to take higher-level math classes. The RAND assessment, one of the largest and most comprehensive studies of its kind to date, used a randomized controlled trial to estimate the effectiveness of CTAI in improving algebra proficiency in a variety of natural school settings, in conditions similar to those of schools that independently adopt it.
To measure the effectiveness of CTAI, the RAND team conducted two parallel experiments, one in middle schools and one in high schools. The study examined these groups separately because the population of students taking algebra in middle schools (grade 8 or earlier) is generally higher-achieving than that of students taking algebra in high schools (grades 9–12), and the curriculum might have different effects in these two contexts.
The setting consisted of 147 school sites: 73 high schools and 74 middle schools in 51 school districts in seven states. Participating schools included urban, suburban, and rural public schools and some Catholic parochial schools. The sites included city districts in Alabama, Connecticut, New Jersey, and Texas; suburban districts near Detroit, Michigan; generally rural districts in Kentucky; and districts throughout Louisiana. Each school participated for two years. Schools were randomized into two groups: an intervention group in which schools received the algebra tutoring curriculum and software, and a control group in which schools instead relied on traditional algebra instruction. Each school included two cohorts of students in the study, corresponding to the schools' first and second years of participation in the study.
Schools from each state participated in both the middle school and high school arms of the study, except Alabama (middle school only). Nearly 18,700 students in grades 9–12 participated in the high school study, with 89 percent of the participants in 9th grade. Nearly 6,800 students in grades 6–8 participated in the middle school study, with more than 99 percent of them in 8th grade.
Changes in students' algebra proficiency were measured with commercial standardized tests. Students took an algebra readiness pretest as a baseline, and then an algebra proficiency posttest after receiving algebra instruction for one year.
The study found that CTAI significantly improved algebra scores for high school students, but the effect emerged in the second year of schools' implementation. Specifically:
What accounts for this large effect size, and why did it occur only in the second year of implementation? Though the research did not identify a rigorous explanation, the team explored two possibilities. The first is that greater teacher experience with CTAI improved student results; however, this possibility was not confirmed by analysis. Researchers divided teachers in the study's second year into two groups, based on whether they had participated in the prior year or were new to using CTAI, and found that student learning was similar for both groups. The second possibility is that all teachers in the second year were better at adapting the CTAI curriculum to meet student needs. Teacher reports of their own practices provide some support to this hypothesis. Further research is needed to test this explanation more rigorously, but it suggests that some patience in seeing results might be warranted when schools adopt innovations.
This large-scale effectiveness trial found a significant positive effect for high schools using CTAI in their second year of implementation, relative to similar schools that continued to use existing textbook-based algebra curricula. This positive result is important for educators and policymakers seeking ways to improve algebra achievement, and it is particularly notable because it was obtained in an effectiveness trial, in which a broad range of schools implemented the curriculum under normal conditions.
The results may also be of broader potential interest because this curriculum uses technology to enable a personalized, blended-learning approach. As one of the first large-scale effectiveness trials of this type of intervention, the results suggest that this may be a productive way to employ technology to improve student achievement in mathematics or other subjects.
The study's results are broadly applicable to schools across the United States, suggesting that high school educators can expect positive results.
Here, the answer is far less certain.
The study found positive effects of CTAI in the second year it was being implemented in schools, but not the first year.
The research team examined the cost of CTAI compared with the more traditional textbook curricula used in control group schools.