Download Free Electronic Document

FormatFile SizeNotes
PDF file 0.2 MB

Use Adobe Acrobat Reader version 7.0 or higher for the best experience.

Research Brief
A female teacher is posed in front of a chalkboard with algebraic equations on it.

Oleksandra Borsuk/Hemera/Thinkstock

Student performance in mathematics remains a source of concern for U.S. educators and policymakers. Although math scores have risen slightly in recent decades, U.S. students still perform poorly on the National Assessment of Educational Progress and in international comparisons with their counterparts from many other countries. In an effort to address this issue, many districts and schools have turned to computer-based tools as a way to boost math performance. These tools allow self-paced instruction and provide students with customized feedback. These features, it is widely held, will improve student engagement and improve proficiency. However, evidence to support these claims remains scarce. In many cases, these tools have been adopted with little or no evaluation.

To make headway in addressing this knowledge gap, a team of RAND researchers assessed whether a popular algebra curriculum that includes tutoring software would be effective in improving the math test scores of middle and high school students. Cognitive Tutor Algebra I (or CTAI), developed by Carnegie Learning, is a first-year algebra course that blends classroom instruction and textbook-based activities with computer-based instruction and has shown efficacy in improving math performance in isolated, small-scale demonstrations. Algebra is of particular interest because it can function as a gateway subject that leads students to take higher-level math classes. The RAND assessment, one of the largest and most comprehensive studies of its kind to date, used a randomized controlled trial to estimate the effectiveness of CTAI in improving algebra proficiency in a variety of natural school settings, in conditions similar to those of schools that independently adopt it.

Approach

To measure the effectiveness of CTAI, the RAND team conducted two parallel experiments, one in middle schools and one in high schools. The study examined these groups separately because the population of students taking algebra in middle schools (grade 8 or earlier) is generally higher-achieving than that of students taking algebra in high schools (grades 9–12), and the curriculum might have different effects in these two contexts.

The setting consisted of 147 school sites: 73 high schools and 74 middle schools in 51 school districts in seven states. Participating schools included urban, suburban, and rural public schools and some Catholic parochial schools. The sites included city districts in Alabama, Connecticut, New Jersey, and Texas; suburban districts near Detroit, Michigan; generally rural districts in Kentucky; and districts throughout Louisiana. Each school participated for two years. Schools were randomized into two groups: an intervention group in which schools received the algebra tutoring curriculum and software, and a control group in which schools instead relied on traditional algebra instruction. Each school included two cohorts of students in the study, corresponding to the schools' first and second years of participation in the study.

Schools from each state participated in both the middle school and high school arms of the study, except Alabama (middle school only). Nearly 18,700 students in grades 9–12 participated in the high school study, with 89 percent of the participants in 9th grade. Nearly 6,800 students in grades 6–8 participated in the middle school study, with more than 99 percent of them in 8th grade.

Changes in students' algebra proficiency were measured with commercial standardized tests. Students took an algebra readiness pretest as a baseline, and then an algebra proficiency posttest after receiving algebra instruction for one year.

Results: CTAI Led to Improvements for High School Students

The study found that CTAI significantly improved algebra scores for high school students, but the effect emerged in the second year of schools' implementation. Specifically:

  • In the first year of implementation, posttest scores for the intervention group schools did not differ significantly from those for the control group schools.
  • In the second year of implementation at the high school level, schools using the CTAI curriculum showed significant improvement in student test scores compared with control-group schools.
    • The effect size — approximately 0.20 standardized effect sizes — is equivalent to moving a student from the 50th percentile on the algebra posttest to the 58th percentile (see Figure 1).
    • This effect is educationally meaningful. The CTAI group outperformed the control group by roughly the amount measured by other studies for mathematics achievement gains from 8th to 9th grade using traditional curricula (see Figure 2).
  • In the second year of implementation at the middle school level, the study estimated a similar-sized effect in favor of CTAI, although it was not statistically significant.
The Effect Is Equivalent to Moving an Algebra I Student from the 50th Percentile on the Algebra Posttest to the 58th Percentile

Figure 1. The Effect Is Equivalent to Moving an Algebra I Student from the 50th Percentile on the Algebra Posttest to the 58th Percentile

a. Students in a traditional algebra course, ranked by their scores on an end-of-year test. Forty-nine students (red) scored lower than the reference score (yellow) and forty-nine students (green) scored higher than the reference score.

b. If these same students had been in the CTAI course, statistical models from the study suggest that they all would perform better and that about eight additional students would score higher than the reference score.

The Effect Is Large Enough to Be Educationally Meaningful

Figure 2. The Effect Is Large Enough to Be Educationally Meaningful

The CTAI group outperformed the control group by roughly the amount measured by other studies for mathematics achievement gains from 8th to 9th grade using traditional curricula.

What accounts for this large effect size, and why did it occur only in the second year of implementation? Though the research did not identify a rigorous explanation, the team explored two possibilities. The first is that greater teacher experience with CTAI improved student results; however, this possibility was not confirmed by analysis. Researchers divided teachers in the study's second year into two groups, based on whether they had participated in the prior year or were new to using CTAI, and found that student learning was similar for both groups. The second possibility is that all teachers in the second year were better at adapting the CTAI curriculum to meet student needs. Teacher reports of their own practices provide some support to this hypothesis. Further research is needed to test this explanation more rigorously, but it suggests that some patience in seeing results might be warranted when schools adopt innovations.

The Bottom Line

This large-scale effectiveness trial found a significant positive effect for high schools using CTAI in their second year of implementation, relative to similar schools that continued to use existing textbook-based algebra curricula. This positive result is important for educators and policymakers seeking ways to improve algebra achievement, and it is particularly notable because it was obtained in an effectiveness trial, in which a broad range of schools implemented the curriculum under normal conditions.

The results may also be of broader potential interest because this curriculum uses technology to enable a personalized, blended-learning approach. As one of the first large-scale effectiveness trials of this type of intervention, the results suggest that this may be a productive way to employ technology to improve student achievement in mathematics or other subjects.

Q&A Guidance for Schools Weighing Use of Cognitive Tutoring Algebra I

Q. Should high schools expect positive results if they adopt CTAI?

The study's results are broadly applicable to schools across the United States, suggesting that high school educators can expect positive results.

  • The study included a cross section of schools across several states, in a variety of geographic locales (urban, suburban, and rural), and it included students with a range of demographic characteristics.
  • To parallel real-world adoptions as closely as possible, there was nothing special about how CTAI was implemented at the study sites.

Q. Should middle schools expect positive results if they adopt CTAI?

Here, the answer is far less certain.

  • Although the middle school study employed the same research design as in high schools and estimated a similar positive effect, the result was not statistically significant.
  • The similarity of the estimated effect suggests that CTAI may produce positive effects in middle schools, but more evidence is needed to confirm or disprove this possibility.

Q. How quickly can schools expect results?

The study found positive effects of CTAI in the second year it was being implemented in schools, but not the first year.

  • The lack of first-year positive effects is consistent with other studies that have found poor results early in the adoptions of innovations.
  • Decisionmakers should be prepared for the possibility that CTAI will not produce positive effects immediately and should allow time for the full effects to emerge.

Q. Is CTAI expensive?

The research team examined the cost of CTAI compared with the more traditional textbook curricula used in control group schools.

  • The team considered all of the costs associated with implementing the curricula, including textbooks, software, technology infrastructure and support, and teacher training.
  • The team found that CTAI is more expensive, estimating it to cost about $97 per student, compared with about $28 per student for the algebra I curricula that were already in place in the control group schools.
  • The specifics of new implementations can cause the costs to vary from these estimates. Decisionmakers need to weigh whether the potential benefits of CTAI are large enough to warrant the cost.

Related Products

Research conducted by

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.