A recent blog post by Max Eden laments the tendency of the education media and researchers alike to succumb to the well-documented inclination of human beings to focus on facts that confirm their prior beliefs. At RAND, we know that this is an immense challenge, not only to human beings in the media but to human researchers as well. That's why we bend over backwards to make sure we present complex findings in the most balanced light possible.
In commenting on a new RAND report about the impact of restorative practices, Eden bemoans “RAND's unmistakably slanted editorial emphases” and quotes a journalist as saying that “[t]he negative test for effect for black kids is buried on like page eighty with no mention (that I saw) until then.”
Following U.S. Department of Education guidance, our report focuses on estimates of the impact of restorative practices on comprehensive outcome measures for all students. We define these measures in seven outcome domains and lead our report with the impact on suspensions, which was the primary objective of the intervention. We also present estimates of the impact on 50 secondary measures as well as on the impact on the seven primary measures for numerous student subgroups defined by race, income, special needs and English language learner status. It is not possible to discuss in depth all of these in the body of the report, much less in the executive summary or research brief targeted at policymakers and practitioners.
Although not mentioned by our critic, we did highlight the presence of negative findings in our reports. We emphasized negative findings in our research brief that was directed to the media, in the executive summary of our larger research report, and in the findings posted on our internet page: “Restorative practices did not positively affect all students in the treatment schools. Despite fewer suspensions, academic outcomes did not improve in the treatment schools. At the middle grade level (grades 6–8), academic outcomes actually worsened in the treatment schools, and suspension rates in those grades did not change…[bold from original]” Furthermore, although the impact on achievement is not the primary focus of the initiative, our presentation of the achievement impact findings is clearly labeled in our table of contents and features a lengthy discussion of the negative impact on African American students.
We definitely agree that every school-based intervention should be in part judged by its impact in achievement, even when achievement is not a focus of the intervention. We also agree that the impact on racial disparities in achievement should always be examined. We also know that many interventions cause a temporary dip in achievement, often seen most starkly among already struggling students, as teachers work out how to incorporate new practices into their routines. Therefore, initial achievement impact estimates should be interpreted cautiously.
In the current study, two hypotheses about the achievement decline for some subgroups immediately come to mind. First, the dip in achievement could reflect time taken away from academic content. Alternatively, suspending fewer students could lead to more disruption in the classroom. Our interpretation of the available data does not provide clear evidence for one or the other of these hypotheses. For some subgroups, such as students in grades 6-8, there was a drop in achievement even though there was no reduction in suspensions. For elementary grade students, there was a drop in suspensions without a drop in achievement. These patterns would suggest that any effects on achievement are not due to the reduction in suspensions. However, for other subgroups, such as African American students and all students in predominantly African American schools, there was a decrease in both suspensions and achievement, which hints that at least in some cases keeping misbehaving students in class might be detrimental to learning. We are continuing to analyze these data and we will release further findings when we are confident in their accuracy.
Another concern expressed by Mr. Eden is that researchers too often blame “poor implementation” for impact findings that they don't like. We agree that the definitive evidence for the value of a program comes from its impact on measurable outcomes, but we also know that change takes time and that implementation evaluation is an invaluable source of information about why a program might not be working as planned. As our report documents, we find clear evidence that students had a more positive view of classroom climate if their teacher reported using specific elements of restorative practices on a regular basis. This, along with teachers' reported views of the program's value and the reduction in suspensions, guided our claim that the program was promising for elementary grade students. We did not claim that the program was a proven solution.
Finally, we are not surprised that commentators with different prior views regarding the value of restorative practices have selected different findings from our study to bolster their opinions. However, we hope and expect policymakers and practitioners to more thoughtfully engage with our findings. Research often results in complex and contradictory findings that are inconsistent with a priori beliefs. Policymakers must wrestle with this complex reality and the apparent contradictions in order to determine whether this is a program that has potential to help the children in their care. We trust that these public servants will join us in the effort to innovate and monitor these programs so that more children can stay in school and all children have the opportunity to learn.
John Engberg is a senior economist at the nonprofit, nonpartisan RAND Corporation. Catherine Augustine is director of RAND's Pittsburgh office, and a senior policy researcher at RAND.
Commentary gives RAND researchers a platform to convey insights based on their professional expertise and often on their peer-reviewed research and analysis.
