on Facebook
on Twitter
on YouTube
RAND Review
Perspectives
High Standards or No Standards?
The Unclear Implications of "No Child Left Behind"
Doomed to "fail"? Third graders from the Lovett School, in Atlanta, gather on the state capitol steps in May 2003 for the announcement of the approval of Georgia’s accountability system aligned with the No Child Left Behind Act. |
WHEN THE NO CHILD LEFT BEHIND (NCLB) Act was passed into law in January 2002, it heralded the beginning of one of the most expansive efforts to reform public education in a generation. Grounded in the need to hold educators responsible for student achievement, NCLB consists of a system of goals (desired student performance), assessments (to measure whether the goals are attained), and incentives (both "carrots" to motivate educators to achieve the goals and "sticks" to punish educators if they don’t).
Although the law passed Congress with overwhelming bipartisan support, concerns have grown—from scattered grumbles in the early years to a groundswell of grievances—as educators begin to grapple with the reality of implementing the law at the state and local levels. Some school districts have filed lawsuits, while other districts have turned down federal money for schools to avoid federal sanctions. Some Republican legislators have complained to the White House that the law violates states’ rights. Growing state pressure has had an effect, which is reflected in recent policy changes by the U.S. Department of Education to make it easier for school districts that have large numbers of students with limited English proficiency to meet their yearly progress goals.
It is within this increasingly contentious environment that Robert L. Linn, professor of education at the University of Colorado and codirector of the National Center for Research on Evaluation, Standards, and Student Testing, came to RAND to talk about his impressions of NCLB.
Deficient Meanings of "Proficient"
Linn argued that performance standards to determine proficiency are essential in many professional cases, such as for licensure and certification, where the content to be tested is fairly straightforward and the goal is to protect the public from unqualified practitioners. But when it comes to setting educational performance standards in terms of proficiency levels—a fundamental premise of NCLB—Linn raised serious concerns.
Linn traced the current drive toward setting proficiency levels to efforts associated with the National Assessment of Educational Progress (NAEP), known as "the nation’s report card," during the 1990s. Driven to set "high standards" by concerns about the United States falling behind other countries educationally, NAEP tests in 1990 defined "proficiency" in mathematics in grades 4, 8, and 12 as scoring at the 87th, 85th, and 88th percentiles, respectively.
Therefore, only 12–15 percent of the students taking the nationally standardized NAEP tests could claim proficiency in 1990. Since its passage in 2002, NCLB has required that increasingly higher proportions of students in each state score at the levels attained by only 12–15 percent of the students in 1990. NCLB requires that fully 100 percent of students attain "proficiency" by the year 2014.
Such high performance standards are supposed to specify "how good is good enough," said Linn. But he argued that such "cut scores," which leave students either falling above or below the "cut" for the designated standard, leave unanswered the question "good enough for what?" What does it really mean, for example, to be "proficient" in mathematics in the fourth grade?
"Incredible" Variability Among States
Understanding what "proficient" means becomes virtually impossible, he continued, because of an even bigger problem: the "incredible" variability both in the starting points that states establish for themselves and in the failure rates, based on the noncomparable "cut" rates that states report for themselves using their own statewide tests.
To begin with, there is variability between what some states have set as their own proficiency levels (on state tests) and their NAEP proficiency levels. For example, Colorado has established a proficiency level for fourth-grade mathematics that designates nearly 80 percent of the students as "proficient," using the standards of the statewide test. However, results from the 2003 NAEP test reveal that only 34 percent of Colorado fourth-graders are proficient in mathematics, using the NAEP definition.
Linn then compared the starting points established by 34 states for fourth-grade reading. He showed that some states, like California, have set very stringent standards for proficiency (where only 14 percent currently meet the statewide standard), while other states, like Colorado, have set very lenient standards (where about 78 percent meet the statewide standard). This 64-percent range across the state test results dwarfs the range of NAEP test results for fourth-grade reading proficiency across all of the states. The national range on the 2003 NAEP tests is only 25 percent.
Finally, Linn said there is enormous variability in the process of measuring proficiency, even when using standardized national tests. For example, failure rates assigned to the same reading tests ranged from 9 to 30 percent among different panels of judges, while failure rates assigned to the same mathematics tests ranged from 14 to 71 percent among different panels of judges. The variability arises partly because the judges themselves—panels of teachers, school administrators, and counselors—have different qualifications.
Different tests also make a difference. Linn referred to a 2003 study that compared the results from NAEP with the results from three other nationally administered tests. Although the NAEP test for fourth-grade reading showed 31 percent of students as proficient, the three other tests scored the national proficiency rates at 24 percent, 40 percent, and 55 percent, respectively.
Balloon Payments Loom
The cornerstone of NCLB is "accountability." Educators not only must set performance standards for basic, proficient, and advanced levels of achievement but must also demonstrate "adequate yearly progress," or AYP, toward meeting those standards by 2014.
Unfortunately, as Linn argued, no state has shown progress rates that would project anything close to becoming 100-percent proficient in 20 years—much less in 10. Once again, there is vast variation across states. In 2003, Kentucky reported that fewer than 5 percent of its schools missed the state’s AYP goals, while Florida reported that nearly 80 percent of its schools missed the AYP goals. This variation has multiple causes, including state demographics, number of grades tested, whether states use confidence intervals, and, of course, how the states define "proficient."
To make matters worse, states have chosen a stairstep approach toward meeting their ultimate AYP goals. Instead of moving incrementally forward (and upward) from their AYP starting points from year to year, states have chosen to move flatly across the same proficiency level for a couple of years and then abruptly up. Linn likened this approach to making no payments on a loan for two years and then having a balloon payment in the third year.
Although there is a legitimate interest in measuring progress in student achievement, Linn concluded that performance standards are not essential for this purpose. He strongly favors other ways to measure progress. One of these ways involves using "norms."
With norms, a fourth grader’s score on a math test, for example, is compared against the average performance of fourth graders on the same test. The average performance is defined as the normal performance. Comparable to "grading on a curve," norms went out of favor with the advent of proficiency standards.
For now, Linn recognized that state and federal laws mandate standards-based reporting and therefore recommended some ways to improve the process. These ways include ensuring that state standard-setters both realize how their standards will be used and understand the comparative information about the different standards used in other states.
Top