Does Reform-Oriented Teaching Make a Difference?

Among the most basic of the many questions that arise in efforts to understand and improve student performance in America's schools is this one: Do specific instructional practices make a difference? The idea that the way teachers teach affects what students learn hardly seems controversial—or even worthy of discussion. For several reasons, however, the question of how—or even whether—students' test scores are related to instructional practices is by no means settled. For example, the large array of different types of teaching complicates efforts to identify the most promising practices, and the characteristics and experiences students bring with them to the classroom also affect learning and therefore must be taken into account.

Recognizing the importance of these issues, a RAND team has examined the relationship between the use of a set of instructional practices called reform-oriented teaching and students' test scores in mathematics and science. The key features of their investigation were

• a longitudinal research design, which enabled the team to assess performance over a three-year period
• a multimethod approach to measuring instructional practices, which included both teachers' reports about their practices and direct observations by the research team
• multiple measures of student performance, which made it possible to determine whether the relationship between teachers' practices and students' test scores varied according to the characteristics of the test.

As might be expected, their findings were complex, but they suggest a weak positive relationship between reform-oriented teaching and test scores on some, but not all, measures of student performance.

These results are important for two reasons. First, the positive relationship between reform-oriented teaching—although weak—suggests that this instructional approach may help students succeed in the mathematics and science courses that were the focus of the study. Second, the differing results obtained on alternative measures of student performance point to the importance of careful consideration of how student performance is measured—both in research and in real-world settings. This document describes those findings and also presents several recommendations that provide a foundation for rigorous research on the relationship between instructional practice and student performance.

Mosaic II: A Detailed Assessment of the Relationship Between Reform-Oriented Teaching and Student Performance

Reform-oriented teaching stresses instruction that engages students as active participants in their own learning as a means of helping them develop complex cognitive skills. As applied in mathematics and science, this approach pushes students to "do math" and "do science," emulating the practices of mathematicians and scientists. In mathematics, for instance, teachers might emphasize the use of manipulative aids, problem-solving, and mathematical communication. In science, students might be expected to generate scientific questions, plan research, collect data, analyze relationships, and write reports.

More generally, reform-oriented instruction encourages inquiry-based activities and the skills of intellectual conversation: asking questions, discussing alternative approaches to problems, presenting reasons for answers to questions, and making connections between old and new knowledge and between superficially disparate topics.

The study of reform-oriented instruction described here, called Mosaic II, was an extension of Mosaic I, an earlier investigation in which a team of RAND researchers found "a weak but positive relationship" between reform-oriented instructional practices and students' scores on standardized tests. Intrigued by these results, they undertook a new study focused on the same question, but designed to yield a more detailed picture of this relationship. The new study involved three key methodological features: a longitudinal research design, multiple measures of instructional practices, and multiple measures of student performance.

Student performance on standardized tests was measured over a three-year period

At the outset of the study, the RAND team selected five cohorts of elementary and middle-school students in three school districts. A cohort consisted of all the students in a selected grade in one of the two subject areas (for example, third-grade mathematics students or sixth-grade science students).^[1] Each cohort was followed for three years, and their performance on standardized tests was measured at the end of each year (current-year performance), as well as aggregated at the end of the three-year period (cumulative performance).

Multiple measures were used to assess instructional practice

To obtain data regarding teachers' choices and classroom activities as they occur, the Mosaic II team used the following methods to assess instructional practices:

• Teacher Surveys: The surveys gathered information about teachers' educational background, experience, and classroom practices.
• Classroom Vignettes: Teachers were presented with written vignettes, consisting of hypothetical classroom situations, and asked to indicate how likely it was that they would choose each of a list of possible actions that could be taken in response. The actions reflected a range of reform-oriented and non-reform-oriented alternatives. Unlike the survey, which required teachers to recall their actions over a wide range of situations over a long period, this method provided teachers with a realistic context in which to situate their responses, potentially offering insights into their preferred teaching style and increasing the generalizability of their responses.
• Teacher Logs: Each day during a two- to five-day period, teachers were asked to fill out a log by checking off items on a list describing specific activities that occurred during mathematics or science lessons. The logs focused on both teacher and student activities each day during the data-collection period. The immediacy of this procedure was intended to increase accuracy in reporting of classroom activities.
• Classroom Observations: The RAND team observed 52 teachers, recording behaviors on a protocol designed to capture important elements of reform-oriented instruction. Because the results of these observations were independent of teacher reports, they were free of any biases due to forgetting or other reporting limitations.

Both multiple-choice and open-ended measures were used to assess student performance on standardized tests

Two measures—scores on both multiple-choice and open-ended items on standardized tests—were used as indicators of student learning in mathematics and science. In addition, performance on items that require problem-solving skills was differentiated from performance on items that required only the application of procedural skills.

Results

Back to our central question: Does reform-oriented teaching improve student performance? In this study, the answer was a tentative "yes."

Exposure to reform-oriented teaching over several years was linked to better cumulative performance in both science and mathematics

In assessments of current-year performance, students who were exposed to a relatively high level of reform-oriented teaching did not generally perform better than students whose teachers were less likely to use these methods. But when cumulative exposure was measured, students who had had more reform-oriented teaching performed slightly better than students who had had less. This was true for both mathematics and science, but the relationship between teaching practices and student performance was slightly stronger in science than in mathematics. Although the association was not large in either subject area, this finding suggests that, as exposure to reform-oriented teaching increases, student performance gradually improves.

In both mathematics and science, the strength of the relationship between reform-oriented teaching and student performance was contingent on how performance was measured

The RAND team found that the strength of the relationship between teaching practices and student performance depended on how performance was measured. Students who had had more exposure to reform-orienting teaching performed better than students who had had less exposure when open-ended measures were used to assess performance, but not when multiple-choice measures were used. This outcome illustrates the importance of considering how student achievement is measured in understanding scores on performance tests, whether the tests are used in research or in practice.

In mathematics, exposure to reform-oriented teaching was more highly correlated with performance on problem-solving items than on items that involved mathematical procedures

In the Mosaic II study, the relationship between exposure to reform-oriented teaching and performance on standardized tests of mathematics varied according to which aspect of mathematical skill was measured, with the positive correlation between exposure to reform-oriented teaching slightly stronger when performance was measured in terms of problem-solving than when measured in terms of ability to perform basic mathematical procedures. This finding is consistent with the underlying premises of reform-oriented teaching, but not entirely so. Although its goal is to promote learning of high-level cognitive skills, such as problem-solving, reform-oriented teaching is also intended to support the learning of procedural skills.

Recommendations

The results of the Mosaic II study provide the basis for several recommendations concerning research on educational reforms. They also have implications for practice.

To capture the effects of education reforms on student learning, choose performance measures that are aligned with the goals of instruction

In this study, different ways of measuring student performance led to somewhat different conclusions about the strength of the relationship between reform-oriented teaching and student performance. In particular, measures that were well aligned with the goals of reform-oriented teaching—that is, open-ended assessments and items that measured problem-solving skills—were most sensitive to variations in level of exposure to reform-oriented instruction.

This finding is likely to hold across educational-reform strategies. That is, the effectiveness of the reform is most likely to be revealed in performance measures that are aligned with the outcomes the reform purports to produce. Choosing tests to assess student performance on this basis might also lead to better implementation of reforms. If teachers understand that a reform they've been asked to implement will be assessed using instruments aligned with the reform goals, they should be motivated to adopt the recommended instructional practices as a means of helping the school meet performance goals. Meeting performance goals, required under the No Child Left Behind Act, might directly affect the status of the school and hence the teacher's working conditions.

Unfortunately, many mandated state and district tests are not currently well aligned with reforms implemented in the schools. Nevertheless, they are being used to assess the effectiveness of the reforms. This study indicates that using these tests, as opposed to tests more aligned with reform goals, might underestimate reform effectiveness or student learning. Because so much testing has been added in recent years, it may be difficult to convince district or school personnel to administer additional tests or to switch to new tests that are better suited to assessing outcomes, given the goals and features of the instruction. This analysis suggests that the use of existing district or state tests may underestimate the full effects of the reforms on student achievement.

To obtain robust, unbiased estimates of the effects of reform-oriented teaching, experimental analyses are needed

Most studies of reform-oriented teaching, including Mosaic II, have been conducted in the context of large national programs. In these situations, researchers are unable to use the most rigorous designs, which include the use of control groups and random assignment. Statistical techniques that adjust for existing differences among students and teachers help to overcome this limitation, but they cannot fully account for unobserved differences. Thus, the next step in the study of the effects of reform-oriented teaching (or other curriculum-reform initiatives) should be randomized experiments. Experiments are sometimes difficult to implement, and they require cooperation from participating districts. But the information that such experiments would yield would almost certainly justify the effort.

Notes

• [1] There were three cohorts in mathematics and two in science. When cohorts were formed in the first year of the study, the mathematics students were in Grades 3–5, 6–8, and 7–9, and science students were in Grades 3–5 and 6–8.