Multimedia

Good morning. This Joe Dougherty and I appreciate that title, but I'm a Media Relations Officer with the RAND Corporation. Thanks for participating in this morning's telephone news conference on the new report from RAND and Mathematica Policy Research, Inc. The study is called Achievement and Attainment in Chicago Charter Schools.

We'll start this morning with an overview of the report. First, coauthor Ron Zimmer, Senior Policy Researcher at RAND, will discuss who's attending charter schools, the impact Chicago charter school attendance has on test scores, and the relationship between attending charter high schools in Chicago and education and payment.

Then coauthor, Brian Gill, a Senior Social Scientist at Mathematica, will discuss the reports' findings for graduation and college entry.

Kevin Booker, a report coauthor, also with Mathematica, is on hand to answer any questions.

When Brian is done, we'll open the discussion for questions. At that time, I'll ask of course that you state your name, the news media organization that you're with, before asking your question.

So thanks again for participating and we'll now begin with Ron Zimmer from RAND. Ron?

Yep. This is Ron Zimmer at RAND. Thank you all for participating today. We're exited about this report. Today we are releasing a report that is a collaboration between researchers at RAND, Mathematica, and Florida State. This is part of a first study to carefully examine whether charter schools have the potential to affect high school graduation and college entry. Therefore, we think it would not only be of an interest to Chicago, but also the rest of the country.

A couple notes I should make up front. This research is funded by a number of different foundations. Two specifically interested in the Chicago portion is the Gates Foundation and the Joy [ph] Foundation.

I should also note that this part of a larger project that is examining charter schools across a number of locations in the United States. This is an early study coming out of this project. There will be a later report that comes out later. What we're going to do there is examine outcomes of charter schools across various locations and see if we can make any links in terms of charter policies or laws in these various locations or at least provide some insights whether these laws are leading to different results.

For today's study, we're focusing on Chicago, and we're examining three questions. We're examining whether charter schools are attracting high or low achieving students, and examining the impact charter schools are having on racial and ethnic mix, we're also examining what impact charter schools have in terms of test scores, and third, we're looking at the relationship between attending a charter high school and educational attainment as defined by ACT scores, graduation, and college attendance.

For all these questions, we are using student level data provided by Chicago public schools.

In terms of the presentation offered, first discuss the first two questions, then I'll turn it over to Brian to discuss the results related to graduation and college entry.

So, the first question:

who is attending charter schools? This is an important question because there's this debate about whether charter schools may be cream skimming the best students from traditional public schools. There is also concerns that charter schools may create greater racial and ethnic stratification.

In the past, many researchers have actually compared the racial mix, for instance, of charters schools, or average racial mix, across the district relative to the district population. However, this may not be the best way to approach this, because charter schools may be clustered in areas of a charter school and be very hard for them to be representative of the whole district because they're not randomly dispersed throughout the district.

In addition, researchers have generally ignored the cream skimming issue altogether. In this particular study, what we have done is look at the longitudinal data in terms of, we have study level, and look at the traditional public schools, students were in, that transferred to charter schools, and look at where they are in terms of the distribution of test scores. Are they high achieving students or low achieving students in the traditional public schools they exited?

What we find there is that charter schools look fairly similar in terms of test score levels to their peers they left. So there's not much evidence here that there's cream skimming going on.

In addition, we looked at the racial and ethnic mix of the traditional public schools that these students exited, and compare that to the charter schools they're going to to see if they're moving to relatively heterogeneous schools to homogeneous schools in terms of race and ethnicity or vice versa. What we're generally finding here, as well, is that there is not much going on in terms of greater racial stratification. They're going to schools that look fairly similar to the schools they exited.

So those are the results on the first question. In terms of the second question, we're looking at what impact does charter school attendance have on test scores. One of the things you really worry about here is that students attending charter schools might be quite different from students attending a traditional public school, because they've made an active choice to attend a charter school. They may be more motivated students or their parents may be more motivated or there could be the opposite, that these students have had a real challenge in traditional public schools and so they're trying charter schools as maybe a last resort. So, it could be also adverse selection as well.

You want to try to control for student self-selecting into these schools. The way we've done it here is examining student achievement gains over time when they're in traditional public schools relative to these same students achievement gains when they're in charter schools. We examine whether they make greater gains in charter schools relative to the gains they made in traditional public schools.

Again, here we're finding not much difference. We're finding that their gains are fairly similar in charter schools to the gains these students made in traditional public schools.

Overall, this indicates that charter schools may be having an affect that's on par with traditional public schools.

Now, I should also mention that was only looking at K-8 school, because we wanted to have the ability to track students from one grade to the next, because we needed to calculate their gains. In Chicago, you only have students tested in consecutive grades in grades 3–8. So that part of the analysis focused on grades 3–8.

Now, I'm going to switch over to Brian to discuss some of the results focusing on the high schools and college attendance and graduations, so I'm switching to Brian now.

Okay, thanks very much, Ron. This is Brian Gill from Mathematica Policy Research. As Ron said, I'm going to talk about the charter high schools in Chicago, and specifically about outcomes associated with attainment.

Here is where we get the stuff that's really novel about this report. As Ron mentioned, at the beginning, this project is really the first research study that has tried to rigorously examine the effects that charter schools have not only on test scores, but also on key attainment outcomes like high school graduation and college entry.

I need to give you a brief introduction to the method we used to analyze those outcomes. As many of you know, as Ron just mentioned, the biggest challenge in determining the effects of charter schools is to distinguish the results that are attributable to the schools from the results that are attributable to the particular kids and families that choose charters.

For assessing achievement test scores, we do that be examining the series of test scores for individual kids when they're in charters and traditional public schools so that we can compare each student to his or her self, to himself or herself. The problem is we can't do that for outcomes like graduation and college entry, because those only happen once. They don't happen repeatedly as tests do.

So we need a different method. To assess the effects of charter high schools on these attainment outcomes, what we did was limited our attention to a group of students, in Chicago, all of whom were in charter schools in 8th grade, just before entering high school. Some of those kids went on to charter high schools; others went to traditional public school in Chicago. The key point is that all of them had chosen charters at some point in the past. Some of them continued in charters and others didn't, but the two groups had nearly identical characteristics in 8th grade before they entered high school.

We make some statistical adjustment for the small differences that remain, those differences actually in terms of test scores. Those baseline differences actually slightly favor the kids who ended up going to traditional public schools, slightly disadvantaged the charter kids.

Then what we do is we follow those two groups, charter 8th graders who went to traditional public schools as a comparison group, charter 8th graders who went to charter high schools as the treatment group. We followed them through high school and beyond, and we find three interesting findings.

For the average charter 8th grader in Chicago, attending a charter high school, is estimated to lead to an advantage of about half a point in composite ACT score. And this is a scale for which the median score is about 16 of the students included in the analysis.

More notably, it leads to an advantage of about 7% points in the probability of graduating from high school, and an advantage of about 11% points in the probability of enrolling in college.

One thing that's important to recognize here is that these results are driven primarily by the Chicago charter high schools that we've called multi-grade high schools. That is, these are schools that actually include middle school grades, and sometimes elementary grades, along side the high school grades. They're schools that have grade configurations that are 6–12, 7–12, or K–12 rather than just 9–12.

So we have to acknowledge here that it's possible that the positive effects we're seeing here for kids enrolled in those charter high schools could be partly a result of the grade configuration itself. It could be that students actually benefit by eliminating the transition from middle school to high school.

That said, there's some uncertainty about how broadly these results will generalize. Are they results that are going to show up in conventionally configured 9–12 charter high schools as well? We're not entirely sure about that. But what is pretty clear is that the multi-grade charter high schools in Chicago appear to be producing attainment effects that are pretty large.

The other thing that's worth mentioning here, as Ron said, this is part of a larger study that's looking at other states. We have graduation and college entry data for charter schools and traditional public schools in the state of Florida as well. We are just doing preliminary work on that and we don't have numbers to report yet. But the preliminary analyses suggest that results are going to be very similarly positive in Florida as they are in Chicago and, moreover, that they may be similarly positive regardless of the grade configuration of the charter high school.

Bottom line here is that the attainment effects we're seeing are, from these multi-grade charter high schools in Chicago, are very promising, quite substantial. This is going to be important not only for policy but also for future research on charter schools.

Studies that look only at test scores without looking at these really important long term attainment outcomes may actually underestimate the full benefits that charter schools have for their students.

I will stop there so that we can open it up to questions. Ron, Kevin Booker, and I are all available to address questions.

Thank you, Ron, and thank you, Brian, for that. I appreciate that. We'll now open it up to questions. Stephanie, can you explain the process on that for us?

Yes, thank you. Ladies and gentlemen, if you have a question at this time, please press the 1 key on your touchtone telephone. If your question has been answered or you wish to remove yourself from the queue, please press the # key.

And our first question comes from Debra Viadero from Education Week. Ma'am, your line is open.

Thank you. Hi, it's Viadero, and I have actually two questions. One is whether the charter schools in the study were smaller than the traditional public schools and whether that was taken into account. And also I'm curious about the size of the sample for that high school level analysis. I think I remember reading in the report that there was kind of high percentage of missing data on kids where you didn't know what happened to them. And I wondered if that was problematic at all.

Good questions. I will start responding to those, and I think Kevin may be able to respond with some additional details.

We don't account for differences in school size in the analysis because we don't think we should do that. As you probably know, charter schools in general tend to be smaller than traditional public schools. But that's actually part of their treatment. They are usually are smaller by design. If that is part of what's going on, it's something that I think appropriately attributed to the charter schools.

I believe it is also — I don't have the numbers right on hand. I believe that general statement about charter schools is probably also true of the Chicago charter schools, although they varied quite a bit in size and Kevin may have some more specific numbers available.

To your second question about the sample size. There are about 1,000 kids included in these high school analyses, the attainment analyses, split pretty evenly across the two groups, kids in charter high schools and kids in traditional public high schools. Yes, there is some attrition. Over time, this is a challenge in any study that tries to look at long-term effects.

We don't think the attrition is too terribly large and we did actually quite a few sensitivity tests to try to see whether that attrition could be biasing the results. We don't find evidence that it is. We made some reasonably conservative assumptions, I think, about the likely effects of the attrition and actually made some predictions about what was likely to happen to kids who disappeared from the data, and they didn't change the results substantially.

The results we end up reporting are those that are slightly more conservative than some that could be used if you hadn't accounted for the difference that attrition is likely to make.

Does that address your questions?

Yes.

Kevin, do we have any — do you happen to have at your fingertips anymore details about the school size issue?

I have don't have the exact numbers. For the multi-grade charter high schools, they serve more than just grades 9–12 and their total enrollment in grades 9–12 tends to be quite smaller than your regular Chicago traditional public high school.

But their overall enrollment in either grades 6–12 or K–12, I'm not sure if that is significantly different.

So thy have the kids distributed across more grades basically and a smaller number of students per grade.

Okay.

We could, if you wanted more details about that we could later dig up some specific numbers.

No, that's helpful. Thanks.

Okay.

Our next question comes from James Brodie from Education Daily. Sir, your line is open.

Thank you. Real quick question, gentlemen. What are some of the characteristics of these schools that distinguish numbers? What are they doing curriculum-wise? What is it that they're doing is different than what the traditional schools are doing?

That is something we'd like to know. This study didn't look at processes at all. It's a study just of outcomes. And we made use of existing data on students, but we didn't have the resources in this effort to actually go into the schools and try to understand what is making the difference there.

I agree that it's a fascinating mixed question, and we would actually love to do some follow up work to try to understand further what could be explaining these effects that we're observing.

This is Ron Zimmer. I just wanted to chime in to say I agree that those are completely valid and important questions, and so we have been trying to follow up with the foundations to say now that we've found these interesting findings, let's try to explore why. So try to get inside the black box.

Ask us again in another year or two.

Okay. I'll hold you to that.

And our next question comes from Bill Zlatos of Pittsburgh Tribune Review. Sir, your line is open.

Thank you. Pittsburgh Public Schools is thinking of moving to multi-grade high schools, 6–12, and I guess hearing the results of the Chicago study, I wondered does this support a performance of charter schools or multi-grade schools?

That's a great question, Bill.

Is this Brian?

Yeah, yeah.

Okay. Thank you.

Good to hear from you again.

Thank you.

Yeah. You know, as I mentioned, this is — one of the ambiguities here is that we can't be 100% sure about how much of these positive effects we're seeing are attributable to charter status per say versus to the grade configuration or to some combination of those things.

And that is something that definitely merits a lot more attention. I am not aware of any studies — we didn't have the data to try to examine this in traditional public schools that had this multi-grade kind of grade configuration.

But it's something that probably could start to be examined in more places. Pittsburgh is not that unusual in going that direction, as you may know. There are other major urban districts that are doing similar things. And I mean what I would say is that the findings here would certainly encourage that kind of experimentation. They can't guarantee that you're going to get the same results from it, but it certainly looks like it's worth looking at.

It's also consistent, by the way, with other research, about the negative effects of switching schools, that eliminating a transition might be a positive thing for kids. So I would say it's something well worth. It certainly supports those efforts even though it can't guarantee that they're going to get the same results.

Thanks, Brian.

And our next question comes from Elizabeth Green from [sic] reporter of New York Sun. Ma'am, your line is open.

Hi. I wanted to ask why you think it might be that the test scores don't have as much information as attainment results. Shouldn't someone who's going to graduated high school and be successful at college score high on tests? And then my second question is about whether — how much it's fair to expand these results to other places? I cover New York City, but I guess anywhere in the country.

Both great questions. I certainly wouldn't argue that the test results are unimportant. I mean they are important, but I think there is lots of evidence that in terms of long term outcomes for kids, graduating and going to college are more important than achievement results, particularly results that are measured in elementary and middle grades.

It's just important to recognize that, as valuable as tests are, they can't tell us everything we need to know about what schools are trying to produce. They are useful, but thin measures of both student outcomes and school performance.

These results are fairly consistent with results that have been seen in other context and studies, for example, of catholic schools where, again, there's certainly arguments about those results but to the extent that they are there, they tend to favor attainment outcomes more than they do test scores.

So, it wouldn't be all that surprising if charter schools are doing things that are valuable for kids in the long term even if they don't — those things don't register on tests.

It, of course, would be useful to follow the kids even further. I mean what we'd really like to know is do they persist in college, do they graduate from college? We need to have a lot more data to be able to follow them that far.

I'd like to chime in as well. This is Ron Zimmer from RAND. A couple things to keep in mind as well. The test scores were focusing on the K–8 schools or grades, and our attainment data in terms of graduation and college enrollment is more focused on high school. So, there could be a hypothesis that the charter high schools are doing a better job.

Now, we can't tease that out, but it may be also that they are just performing differently.

Yeah. That's important to mention, especially given that one thing to notice here is that in the high schools, we actually do see a positive evidence of a positive charter effect for ACT scores. So it's not only for graduation and college entry, but also on an important college entry test.

And the second thing I would like to mention here is there was a study, I think, by the Chicago Consortium, maybe a year ago or so, that showed that students with similar ACT scores, in different schools, weren't necessarily all going to college. Some wouldn't go on to college and some would, and some would go to different types of schools. So, it may be also just an environment which the high schools are creating in terms of expectations of going to college or the counseling they're getting. There's lots of hypotheses we could create here about why the difference. But, at this point, it's a speculation.

Let me turn to your second question about what does this say about other places. It's very hard to know yet. As we said, this is the first study that has attempted to look seriously at attainment outcomes associated with charter schools. We see very promising results in Chicago. It looks like we're also going to see promising results in Florida. That report I hope will be out within the next couple of months.

But even then, at most we've got two sites where it's been possible to look at this. Whether it will be replicated in other parts of the country is, I don't know. It, again, it certainly suggests promise for charter schools. And it strongly suggests that both for policy makers and researchers all over the country, we ought to be looking at those outcomes for charter schools and not only at test scores.

What I'm saying, this is Ron Zimmer again, I should mention is the real challenge of doing that is that to link K-12 data to college attainment data, not many places do that. I think only a handful of places across the country and so maybe this is something that policy makers want to think about whether they want to try to make a greater effort of linking these type of data.

Yeah, that's right. There is something called the National Student Clearing House which has college records for over 90% of the institutions of higher education in the country. And this is where the Chicago Public Schools gets their data on college entry. I think other cities and school districts and states across the country would do well to work with the Clearing House so that they can link their data on K12 systems to, and follow those kids into college.

Thanks.

And our next question comes from Debra Viadero from Education Week. Ma'am, your line is open.

Hi, again. I'd like to hear some more about the larger project of which this is a part and also just to follow up a little bit more on Elizabeth's question. I mean how similar are the charter schools in Chicago to other schools around — other charter programs around the country, how typical?

Ron, do you want to (inaudible)?

Yeah, I can speak to this first question, this is Ron Zimmer from RAND again, in terms of the larger project. So we are looking at charter schools in a number of states. I can hopefully list off all the locations:

San Diego, California; Denver, Colorado; the state of Texas; the state of Ohio; the state of Florida; Milwaukee, Wisconsin; Chicago, Illinois; and Philadelphia, Pennsylvania.

And so we're going to try to look at — and only two locations, Florida and Chicago, are we going to be looking at attainment because that's the only places we have college attendance.

But in other locations we're going to try to look at test scores and also this racial mix and cream skimming question. We hope to kind of see if there are different outcomes across these different locations, can we gain insights about what charter policies and laws are in place in hopes to inform policy makers about these laws — the ways the laws are structured, might lead to different results.

In terms of how Chicago might be different from other locations, you know, Brian and Kevin can chime in as well, but Chicago obviously is a big urban school district and charter schools are often located in big urban school districts, but they're also located in lots of other types of districts. Also, I think Chicago might be a little different than some other locations in the sense that they've received probably greater support from the school district than other locations. They seem to be working in collaboration more with the district than possibly many other districts across the country. So that might be different as well.

But those are some things that come to my mind initially. Maybe Brian can chime in as well.

Yeah. This is Brian Gill. Yeah, I would say the same thing that Ron just said. I mean, you know, the larger project we're undertaking here is intended to try to address some aspects of your second question that we want to understand whether there are important differences in charter school outcomes across different states and different laws and policy regimes.

To try to understand whether there are policy levers that might contribute to better outcomes, I don't think we have a good sense of that yet at this point in the project.

How long is the project?

We are hoping to have a final report on all of our sites, Ron?

By the end of the calendar year.

Okay.

Oh, okay.

The Chicago report is actually the second one. We've already released a report on Philadelphia. That one did not include attainment outcomes, because we don't have attainment data there.

But I guess it's fair to say that the achievement — the test score outcomes in Chicago and Philadelphia were not dramatically different, that in both places the charter schools were doing about as well as traditional public schools.

That is also consistent with a lot of the other studies of charter school test results. Prior studies in Texas, in Florida --

Carolina.

Right, so. So the test score results in Chicago are pretty similar to what most of the studies in other sites have seen. We can't compare the attainment results, because they don't exist anywhere else.

Thanks.

Other questions.

Our next question comes from Nancy Zuckerbrod, a journalist. Ma'am, your line is open.

Hi. This Nancy from AP. I just have a question about, possibly for looking forward. Explain student mobility and what we know about student mobility and its impact on kids, and how you think sort of the research that's out there and how you think this adds to that body of research.

This is Ron Zimmer. There has been work, I think, by Eric Hamachek [ph] in Texas, has looked at student mobility and found that it had an negative effect on students. So, on transferring from one school to another seems to have an adverse effect on test scores.

In terms of here, you know, I think it's only a hypothesis right now that eliminating the transition between 8th and 9th grade might be helpful. There could be other explanations of what's going on here, but that could be one explanation of what's going on.

I should also note, in Philadelphia we looked at mobility a little bit in terms of whether charter schools had greater mobility than traditional public schools because you might think they could because it's easier to exist and enter into charter schools, but we didn't find much difference in mobility.

Thanks.

Yeah. This Brian Gill. I suppose I would agree with everything Ron just said and add that, you know, ‘kids who are mobile tend to do worse,' it's a little hard to know whether that's because they are mobile or because mobility is often an indictor for other kinds of instability and family difficulties.

From that perspective, it makes it difficult to know whether simply eliminating a transition would help, but it's certainly possible that the transitions make things more challenging. 9th grade is a very difficult year for a lot of kids and it could be that keeping them in the same schools would help.

Another hypothesis one might put forward is the possibility that having a school that includes multiple grades increases the collaboration and kind of accountability among teachers across grade levels. When the 8th grade teachers and the 9th grade teachers are part of the same school rather than a separate school, it may be easier for them to communicate and collaborate and keep each other informed about particular needs of their kids.

Our next question comes from Sarah Carp of Catalyst Magazine. Ma'am, your line is open.

Hi. I just wanted to ask about whether you'd looked at all the grades of the kids coming out of 8th grade and if that had any difference on where they went to high school, whether they stayed in the charter or went to a different school. And then I was also wondering if you looked at what type of neighborhood high — like if they went to a traditional neighborhood high school or is they went to a magnet school or selective schools.

Yeah. Good questions. We don't have data on the course grades of kids. Kevin, is that right?

That is right.

Okay. And then in terms of the particular traditional public schools that kids went into, as you probably know, Chicago has a variety of different kinds of high schools that are district run sort of the in the traditional public school group that might be traditional or, in fact, untraditional in terms of their educational approaches. They have some magnet schools, some of which are academically selective.

The analysis we did here includes everywhere the kids went, basically any traditional public high school in Chicago. To the extent that those kids, in the sample, went on to academically selective high schools there include here, we didn't split out any of the high schools as a separate treatment group because we wanted to have a comparison of charter schools to wherever the kids might have otherwise ended up.

Kevin, is there anything to add to that? Is that basically --

That's correct.

Okay.

Thank you.

And our next question comes from Elizabeth Green, reporter of New York Sun. Ma'am, your line is open.

Hi. I just had another question about — following up on the idea of how students got out of charter schools. Was there any difference between the kinds of charter schools did better or worse that you found? And do you think that choice played any role in having a choice in where you go to school?

Brian Gill from Mathematica again.

Hi.

I wish we could say something more about the kinds of charter schools that make a difference here. Again, this study didn't have the resources to actually go into the schools and try to understand what they were doing that might have had this effect and we'd love to follow up to learn more about that.

If there was some way to do it in terms of kinds of schools, the challenge here is that it's not a very large number of charter schools we're looking at and so we wouldn't be able to cut the sample very much and say anything very definitive.

So, this is something that's going to have to be done, I think — it might be something we can look at in a little more depth in Florida where we've got the whole state and we can look at a larger number of schools and maybe there are distinctions we'll be able to tease out. Although, again, is limited by the extent to which we've actually got data on those.

Okay, now I've forgotten the second part of your question.

Choice.

Oh, yes. Well, so choice, you know, as I said at the beginning, the fact of choice creates the biggest methodological challenge for researchers in trying to understand — are these effects of the schools or is just telling us something about these kids are different because they chose?

We think that on that end, the method we've used here is likely to be pretty effective because all of the kids had chosen charter schools prior to high school. They were all in charter schools in 8th grade. And so we think we have kind of neutralized the, as much as possible, whether choice might be producing this result sort of independently of the charter schools themselves.

Now, there's another way to look at your question though, which is kind of:

does a policy regime that allows a choice of schools permit this to happen? And if that's what you got in mind, then I guess we'd have to say that's part of the intervention. Yes, that's what charter schools are about is creating more choices.

And so to that extent, yes, you know, it's possible that the charter schools are getting good results because they allow kids and families to find a better match to their particular needs. We don't know if that's the mechanism, but it could be. Or at least could part of it.

I'm not sure if I got at what you were getting at there.

That's, yeah. I mean I guess the kids that leave 8th grade could maybe they went to — I don't know Chicago well, but they maybe went to a worse charter school; maybe it's more ideal and you try harder to get into a multi-grade charter school.

Oh, I see. Yeah. I mean we don't have a randomized experiment here so we can never know with 100% certainty that we fully — that these two groups are really the same in important respects.

But one thing we can tell you. If you look at the report, you'll see there's a table in there that shows you the characteristics of the kids as of 8th grade before they split up, essentially before they went on to charter high schools or traditional public high schools, and the two groups that were about to go on to different sets of schools, looked nearly identical in terms of their demographic characteristics in terms of poverty, race, ethnicity. The only notable difference that these two groups had was that the kids who ended up in traditional public schools were actually slightly higher achieving in 8th grade.

So, if anything, you would expect that to cut the other direction.

Okay, thanks.

Thank you, Stephanie, for handling that. Thank you Ron, thank you Brian. Joe Dougherty here with RAND Media Relations. If you would like to follow up with any questions about any specifics you might have, feel free to send an email to media@rand.org if you'd like to set up a separate interview with either Ron or Brian.

So thank you again for attending and for participating in the conference. Of course, you can find everything online as well at rand.org.

Thanks and have a great morning.