commentary

(San Diego Union-Tribune)

September 19, 2004

Navigating Research on Alternative Schools

by Ron Zimmer and Christopher Nelson

Just about all parents want their children to go to outstanding schools and get the best education possible. Not too long ago, you had to live in a top public school district – where housing prices are often high – or be able to afford private school tuition to achieve this goal. But today, public charter schools, voucher programs and magnet schools allow parents to change schools without moving to a new neighborhood or facing high tuition bills. In addition, the federal No Child Left Behind Act allows parents to opt out of schools that have failed to make adequate achievement progress for two consecutive years.

This ability to choose schools for your children might bring benefits – but at the same time can be stressful. The best way to make the right choice about sending your child to an alternative school is to educate yourself about what's available and what's most effective, so you can make an informed choice. But how can parents make informed choices when news reports are so often filled with contradictory claims about how much certain types of schools can achieve?

Here are our suggestions, as educational researchers and parents, on how best to use the research to make the best school choices for your children.

Take the long view. Instead of focusing on just the latest news story about a new study of a certain type of school or teaching method, look at a lot of studies dealing with the same subject. Consider what the totality of the evidence indicates rather than a finding from a particular study. If a number of studies suggest that an educational program is effective, that's often a strong indication that the program is a good one. But if different studies arrive at different conclusions, it's often safest to assume that little is known.

The challenge of interpreting competing findings is illustrated by an Aug. 17 New York Times article headlined "Nation's Charter Schools Lagging Behind, U.S. Test Scores Reveal." This article raised concerns about the performance of charter schools, which are independently operated public schools. But since the publication of the news story, a number of articles have appeared in newspapers outlining the weaknesses of the research cited by the Times, and have drawn attention to research that counters conclusions found in the research in the Times. As analyses by the RAND Corporation and others have shown, on the whole the evidence base on charter schools remains weak and presents a very mixed picture of the schools' academic effectiveness.

Consider scientific merit. All educational studies are not created equal. Much of the published research on charter schools and school choice is not subject to rigorous scientific peer review. So parents need to become savvy about distinguishing high quality from low quality studies. This is often difficult, especially with studies that consider questions of student achievement. Here, the issues tend to be complex and highly technical. One way to identify a credible study is to see if it clearly describes how researchers arrived at their findings. Studies without discussions of methods should be viewed with suspicion.

Question what's responsible for good student performance. Parents should try to figure out if students at a particular school or type of school – such as charter or magnet schools – are doing well because of the education they are receiving, or because of the abilities they had when they enrolled. Many studies erroneously conclude that if students in alternative schools score higher on achievement tests than students in standard public schools, the alternative schools must be more effective in promoting learning. In fact, the test results might just tell us that high performing students are more likely to be attracted to certain alternative schools. For example, a magnet or charter school that emphasizes advanced math instruction is likely to draw some of the best math students to enroll, and those students are likely to score well on math tests. A far better way to assess school effectiveness is to see if students in a school make big individual improvements in their educational performance over time. Unfortunately, student data systems in many states make such research difficult because the systems do not allow researchers to track the achievement gains of individual students over several years. Such information should become more widely available as states gear up for testing requirements of the federal No Child Left Behind Act.

Proving that a charter or other alternative school is responsible for higher achievement also requires comparisons with conventional schools educating similar students. Otherwise, it is impossible to know whether the observed achievement gains would have occurred had the students attended a different kind of school. The best comparison groups are created through random assignment to a treatment and control group (similar to a medical study). Where this is not possible, studies can match alternative schools with demographically similar conventional schools. Unfortunately, this approach is not as good as randomized groups, since some of the most important differences between alternative and conventional school students often stem from difficult-to-measure factors such as motivation and parental support. But even imperfect controls are better than no controls.

Consider the source. Commonly, disagreements over research are laced with charges and countercharges of researcher bias. Increasingly, advocacy groups sponsor, conduct and issue studies. Bias is a legitimate concern here, and parents should give the most weight to educational studies conducted by independent, nonpartisan organizations. In the end, though, it is better to evaluate research on its scientific merits, such as the quality of the control group used.

Think about what you value in a school. Test scores are often just one of many factors to consider. Other criteria may include safety at the school, the diversity of the students, the curriculum, instructional practices, or the overall educational philosophy of the school. Schools that are strong in one area might not be strong in others. In fact, studies of charter schools in Michigan and Pennsylvania found little relationship between achievement and how satisfied parents were with instruction and the school curriculum.

Understand variations among schools. Education research is usually designed to help policy-makers evaluate a type of school or program. So researchers might examine dozens or even hundreds of charter schools to gauge the overall performance of charters. But what is true for a large group of schools on average is not necessarily true for every single school in the group. For example, a recent RAND study of California charter schools found that, on the whole, charter school performance roughly equaled that of similar conventional schools. However, a closer look at the data revealed that the findings differed for three different kinds of charter schools: start-up charters, converted conventional public schools, and non-classroom-based schools.

Consider your child's individual needs. As in medicine, scientific research on schools can often provide the general odds that a program or reform will succeed under certain circumstances. As in medicine, though, it's important to avoid blind and unreflective applications of research findings. Good physicians often ask their patients to consider more personal issues such as their general health, age, or lifestyle preferences. Patients can then reflect on this information, talk to other people who have had the treatment, and search the Internet for information to make their decision. Similarly, parents need to supplement research findings with: their own Internet searches; conversations with their children, other parents, principals and teachers; and reflections on the strengths and preferences of their child.

Be realistic about what research can and cannot do. Making good choices for your children is not easy. Scientific research on education can assist these choices. But the technical nature of much research often makes it difficult for parents to distinguish good from bad studies. Keeping in mind the points above can help parents navigate the maze of claims and counterclaims about alternative schools. Moreover, they must learn to integrate research findings with more informal information gathering. Research can ease the burdens of making good choices. But it cannot eliminate that burden.

Zimmer and Nelson are education researchers at the RAND Corporation, a nonprofit research organization.

This commentary originally appeared in San Diego Union-Tribune on September 19, 2004.