Improving Mathematics and Science Education: A Longitudinal Investigation of the Relationship Between Reform-Oriented Instruction and Student Achievement
Sep 8, 2006
In a three-year study, RAND researchers examined the relationship between reform-oriented instruction and student performance in mathematics and science. At the end of the study, students who had been exposed to more reform-oriented teaching performed better in both math and science than those who had experienced less, but the differences in scores were small. The relationship between instructional practice and student performance was stronger when performance was measured on items that required problem-solving skills rather than procedural skills, an outcome that is generally consistent with the goals of reform-oriented teaching. These results illustrate the importance of matching performance measures to reform goals in evaluating instructional innovations.
Among the most basic of the many questions that arise in efforts to understand and improve student performance in America's schools is this one: Do specific instructional practices make a difference? The idea that the way teachers teach affects what students learn hardly seems controversial—or even worthy of discussion. For several reasons, however, the question of how—or even whether—students' test scores are related to instructional practices is by no means settled. For example, the large array of different types of teaching complicates efforts to identify the most promising practices, and the characteristics and experiences students bring with them to the classroom also affect learning and therefore must be taken into account.
Recognizing the importance of these issues, a RAND team has examined the relationship between the use of a set of instructional practices called reform-oriented teaching and students' test scores in mathematics and science. The key features of their investigation were
As might be expected, their findings were complex, but they suggest a weak positive relationship between reform-oriented teaching and test scores on some, but not all, measures of student performance.
These results are important for two reasons. First, the positive relationship between reform-oriented teaching—although weak—suggests that this instructional approach may help students succeed in the mathematics and science courses that were the focus of the study. Second, the differing results obtained on alternative measures of student performance point to the importance of careful consideration of how student performance is measured—both in research and in real-world settings. This document describes those findings and also presents several recommendations that provide a foundation for rigorous research on the relationship between instructional practice and student performance.
Reform-oriented teaching stresses instruction that engages students as active participants in their own learning as a means of helping them develop complex cognitive skills. As applied in mathematics and science, this approach pushes students to "do math" and "do science," emulating the practices of mathematicians and scientists. In mathematics, for instance, teachers might emphasize the use of manipulative aids, problem-solving, and mathematical communication. In science, students might be expected to generate scientific questions, plan research, collect data, analyze relationships, and write reports.
More generally, reform-oriented instruction encourages inquiry-based activities and the skills of intellectual conversation: asking questions, discussing alternative approaches to problems, presenting reasons for answers to questions, and making connections between old and new knowledge and between superficially disparate topics.
The study of reform-oriented instruction described here, called Mosaic II, was an extension of Mosaic I, an earlier investigation in which a team of RAND researchers found "a weak but positive relationship" between reform-oriented instructional practices and students' scores on standardized tests. Intrigued by these results, they undertook a new study focused on the same question, but designed to yield a more detailed picture of this relationship. The new study involved three key methodological features: a longitudinal research design, multiple measures of instructional practices, and multiple measures of student performance.
At the outset of the study, the RAND team selected five cohorts of elementary and middle-school students in three school districts. A cohort consisted of all the students in a selected grade in one of the two subject areas (for example, third-grade mathematics students or sixth-grade science students). Each cohort was followed for three years, and their performance on standardized tests was measured at the end of each year (current-year performance), as well as aggregated at the end of the three-year period (cumulative performance).
To obtain data regarding teachers' choices and classroom activities as they occur, the Mosaic II team used the following methods to assess instructional practices:
Two measures—scores on both multiple-choice and open-ended items on standardized tests—were used as indicators of student learning in mathematics and science. In addition, performance on items that require problem-solving skills was differentiated from performance on items that required only the application of procedural skills.
Back to our central question: Does reform-oriented teaching improve student performance? In this study, the answer was a tentative "yes."
In assessments of current-year performance, students who were exposed to a relatively high level of reform-oriented teaching did not generally perform better than students whose teachers were less likely to use these methods. But when cumulative exposure was measured, students who had had more reform-oriented teaching performed slightly better than students who had had less. This was true for both mathematics and science, but the relationship between teaching practices and student performance was slightly stronger in science than in mathematics. Although the association was not large in either subject area, this finding suggests that, as exposure to reform-oriented teaching increases, student performance gradually improves.
The RAND team found that the strength of the relationship between teaching practices and student performance depended on how performance was measured. Students who had had more exposure to reform-orienting teaching performed better than students who had had less exposure when open-ended measures were used to assess performance, but not when multiple-choice measures were used. This outcome illustrates the importance of considering how student achievement is measured in understanding scores on performance tests, whether the tests are used in research or in practice.
In the Mosaic II study, the relationship between exposure to reform-oriented teaching and performance on standardized tests of mathematics varied according to which aspect of mathematical skill was measured, with the positive correlation between exposure to reform-oriented teaching slightly stronger when performance was measured in terms of problem-solving than when measured in terms of ability to perform basic mathematical procedures. This finding is consistent with the underlying premises of reform-oriented teaching, but not entirely so. Although its goal is to promote learning of high-level cognitive skills, such as problem-solving, reform-oriented teaching is also intended to support the learning of procedural skills.
The results of the Mosaic II study provide the basis for several recommendations concerning research on educational reforms. They also have implications for practice.
In this study, different ways of measuring student performance led to somewhat different conclusions about the strength of the relationship between reform-oriented teaching and student performance. In particular, measures that were well aligned with the goals of reform-oriented teaching—that is, open-ended assessments and items that measured problem-solving skills—were most sensitive to variations in level of exposure to reform-oriented instruction.
This finding is likely to hold across educational-reform strategies. That is, the effectiveness of the reform is most likely to be revealed in performance measures that are aligned with the outcomes the reform purports to produce. Choosing tests to assess student performance on this basis might also lead to better implementation of reforms. If teachers understand that a reform they've been asked to implement will be assessed using instruments aligned with the reform goals, they should be motivated to adopt the recommended instructional practices as a means of helping the school meet performance goals. Meeting performance goals, required under the No Child Left Behind Act, might directly affect the status of the school and hence the teacher's working conditions.
Unfortunately, many mandated state and district tests are not currently well aligned with reforms implemented in the schools. Nevertheless, they are being used to assess the effectiveness of the reforms. This study indicates that using these tests, as opposed to tests more aligned with reform goals, might underestimate reform effectiveness or student learning. Because so much testing has been added in recent years, it may be difficult to convince district or school personnel to administer additional tests or to switch to new tests that are better suited to assessing outcomes, given the goals and features of the instruction. This analysis suggests that the use of existing district or state tests may underestimate the full effects of the reforms on student achievement.
Most studies of reform-oriented teaching, including Mosaic II, have been conducted in the context of large national programs. In these situations, researchers are unable to use the most rigorous designs, which include the use of control groups and random assignment. Statistical techniques that adjust for existing differences among students and teachers help to overcome this limitation, but they cannot fully account for unobserved differences. Thus, the next step in the study of the effects of reform-oriented teaching (or other curriculum-reform initiatives) should be randomized experiments. Experiments are sometimes difficult to implement, and they require cooperation from participating districts. But the information that such experiments would yield would almost certainly justify the effort.