Link to a Statement of RAND President and CEO, James A. Thomson

The team's report, an issue paper titled What Do Test Scores in Texas Tell Us?, raises "serious questions" about the validity of those gains. It also cautions about the danger of making decisions to sanction or reward students, teachers and schools on the basis of test scores that may be inflated or misleading. Finally, it suggests some steps that states can take to increase the likelihood that their test results merit public confidence and provide a sound basis for educational policy.

To investigate whether the dramatic math and reading gains on the TAAS represent actual academic progress, the researchers compared these gains to score changes in Texas on another test, the National Assessment of Educational Progress (NAEP). The NAEP tests were used as a benchmark because they reflect standards endorsed by a national panel of experts, they are not subject to pressures to boost scores, and they are generally considered the nation's single best indicator of student achievement. Both the TAAS and the NAEP tests were administered to fourth and eighth graders during comparable four-year periods.

The RAND team--Stephen P. Klein, Laura Hamilton, Daniel McCaffrey and Brian M. Stecher--generally found only small increases, similar to those observed nationwide, in the Texas NAEP scores. Meanwhile, the TAAS scores were soaring. Texas students did improve significantly more on a fourth-grade NAEP math test than their counterparts nationally. But again, the size of this gain was smaller than their gains on TAAS and was not present on the eighth-grade math test.

The "stark differences" between the stories told by NAEP and TAAS are especially striking when it comes to the gap in average scores between whites and students of color. According to the NAEP results, that gap in Texas is not only very large but increasing slightly. According to TAAS scores, the gap is much smaller and decreasing greatly.

"We do not know the source of these differences," the researchers state. But one reasonable explanation, consistent with survey and observation data, is that "many schools are devoting a great deal of class time to highly specific TAAS preparation." While this preparation may improve TAAS scores, it may not help students develop necessary reading and math skills. The authors suspect that "schools with relatively large percentages of minority and poor students may be doing this more than other schools." Other features of the TAAS also may contribute to the false sense that the racial gaps are closing.

Problems with statewide tests are not confined to the TAAS or Texas, the authors observe. To lessen the likelihood of invalid scores on such tests, they recommend that states:

• Reduce the pressure associated with high-stakes testing by using one set of measures for decisions about individual students and another set for teachers and schools.

• Replace traditional paper-and-pencil multiple choice exams with computer-based tests that are delivered over the Internet and draw on banks of thousands of questions.

• Periodically conduct audit testing to validate score gains.

• Examine the positive and negative effects of the testing programs on curriculum and instruction.

RAND is a nonprofit organization that helps improve policy and decisionmaking through research and analysis.