The Rise of Standards-Aligned Materials for English Language Arts and Math Instruction

Transcript

Julia Kaufman

Good afternoon, everyone. We're so pleased that you could join us, so excited to be here today to talk with you about findings from our latest report, which was just out today, and was authored by me, my co-presenter, Sy Doan, and our wonderful colleague Maria-Paz Fernandez, who worked with RAND this summer as a summer associate and is a graduate student at UCLA School of Education and Information Studies. I'm even more excited that this is our first joint webinar ever with EdReports, who has been so instrumental in making much of our research possible. So I've introduced myself. I'm a senior policy researcher at RAND and co-director of the RAND American Educator Panels. I'd love for my co-presenters to introduce themselves. Sy?

Sy Doan

Hi everyone, I'm Sy Doan. I'm an associate policy researcher at the RAND Corporation, and I've been a project researcher for the past three years on the American Instructional Resources Survey (AIRS), and I'm really excited to share all the new data that we have with all of you today.

Julia Kaufman

And, Eric.

Eric Hirsch

Hey, there. I'm Eric Hirsch, I'm the executive director of EdReports, all the way back since our founding in 2014. And also excited to have my colleague, Mark LaVenia, who is going to help us with the Q&A and chat, and is our impact specialist.

Julia Kaufman

Thanks, both of you. So let me give you a rundown of what we're going to talk about today. So we're going to talk a little bit about why materials matter. Some of you might not need that discussion, but we're going to share it with you. And then we'll talk a little bit more about the EdReports and RAND work that we've been doing to date to understand standards-aligned materials and their use. And then we'll walk through some findings from our latest survey, which was fielded back in the Spring. We'll talk about teachers' reports on what they're using in the classroom, and we'll also talk a little bit about how instructional materials use has been going during the pandemic over the last school year. Lastly, we'll get into some of the factors that we think maybe influencing what teachers say they use. We'll talk a little bit about the IMPD network, which stands for the High-Quality Instructional Materials and Professional Development network that is part of the Council of Chief State School Officers. And it's a network of states that are working on reforms, policies, laws that can support use of standards-aligned materials across their states. We'll also talk about the presence of EdReports as another factor that may be influencing use of standards-aligned materials, as well as other factors within the control of schools and school systems.

So before we get into that, we'd love to know who is in the room. So you'll see a poll pop up in a minute, and we'll give you a minute to tell us who you are. If you're getting this poll, please let us know where you're coming from. Are you an educator, a school or district leader, a researcher, a policy maker, staff from an organization that supports K-12 educators and leaders, staff from an organization that develops instructional materials, or someone else entirely? So we'll give you a few seconds to do that, and then we'll take a look at the results.

So, a great mix. This is wonderful. All right. That's really helpful, and will help us as we're going through our results today.

So I'm going to talk a little bit about materials. It makes sense that instructional materials teachers use in the classroom will make a difference for what students learn. We've known a long time that curriculum can make a difference for the content that is taught and how it is taught, but we know that from a lot of disparate studies. In addition, several rigorous studies, quasi-experimental studies, have identified curricula that produce gains in student achievement between 10 and 25 percentile points, depending on the study. And some studies of states and what state-approved curricula is being used have identified gains of around 5 percentile points for adoption of a given curricula.

At the same time, a recent study out of Harvard led by David Lazer and others looked at the adopted textbooks in schools across six states and particularly looked at those materials in math that are standards aligned. And they found no differences in math achievement based on the textbook that had been adopted by the school.

So what does this all mean? We think that it means that while curriculum matters, curriculum effects may depend on a lot of factors that go way beyond just adoption of certain curricula. In our own studies here at RAND, we have found ways in which the adoption of standards-aligned curricula is likely related to what teachers know and do, as well as what school leaders know. For example, in some of our recent reports, we found that teachers who report using at least one standards-aligned math material reported higher knowledge of math so knew what content was standards-aligned at each grade level. And they also reported more standards-aligned instruction, kind of engaging in things like telling us that students were persevering in working on problem-solving, and the sorts of things that we know are aligned with standards for mathematics. U.S. school leaders have also reported that they know more about reading approaches aligned with their standards when they report they're using at least one standards-aligned material.

So now I'm going to get into our American Instructional Resources Survey. We've been surveying teachers and school leaders across the United States and in specific states in regard to what material they're using in the classroom for the past three years. We've been doing that through the American Educator Panels. So the RAND American Educator Panels consist of the American Teacher Panel, the American School Leader Panel, and the American School District Leader Panel. These are educators and administrators who have agreed to respond to surveys periodically. And we've put out a lot of reports over the past year, especially related to the pandemic and other matters. We regularly check in with some of the same teachers, but we also have a big pool of 25,000 teachers in our American Teacher Panel and 7,500 school leaders who we survey regularly. And so we survey samples of that larger sample on a regular basis.

Several features of our panels differentiate them from other national surveys. First, we randomly select participants across the United States. We reach out to them and ask them if they would be willing to serve on the panel. That includes those from lots of different types of schools, including low- and high-income schools, varying levels of experience. And then after we conduct surveys, we also weight the results so that they're nationally representative. We give participants an honoraria based on survey length. And again, they get several surveys a year.

We also have a data portal, and so anytime we do a survey, we post the data in our data portal. Sometimes it takes a little while for the data to come up on our data portal. But we do have available data for free that folks can download. We also have wonderful partners at Kitamba, who've provided us with a free data visualization tool for exploring these data. The AIRS 2021 data will be loaded onto Bento in, I think, November, and the AIRS data from past administrations are already on Bento and you can explore them right now.

So let me talk a little bit more about the American Instructional Resources Survey. It focuses on what instructional materials are used in schools, how and why materials are used, and what could be supporting people to use those materials. AIRS was fielded to nationally representative samples of teachers and principals starting in Spring 2019, and so today we'll be bringing in the Spring 2021 data. We'll be doing a little comparison to the Spring 2019 and 2020 data. The AIRS was also fielded to state representative samples in multiple states, including those states that are part of the IMPD network.

All right. Well, I'm going to pass it on to Eric so he can talk a little bit about EdReports work.

Eric Hirsch

Great. Thank you so much, Julia and Sy for having us. Knowing that EdReports is such an important barometer for you as we think about standards-aligned material, and we hope for the field, we wanted to take a second to make sure the folks on the webinar and in general know who we are and how we do our work. So since 2014, EdReports has been working with educators to review core instructional materials, at the time in math, but now math, ELA, and science, in order to empower districts and states with evidence and information, in particular around standards alignment and usability.

And we really grew out of the notion of the Common Core and other college and career-ready standards that were quite different in states—they were more rigorous; they were more focused; they required different things of students and teachers. Yet within days there were materials that were claiming alignment to these standards, and our hope is by going in and working with educators who are with kids every day and using these materials, we'll be able to inform that conversation.

Our theory of action is really about writing these reports. Ultimately, and I hope you've all been on the website and looked at our over 900 grade and course level reports across different content areas, that, again, if we identify standards alignment, usability, and other aspects of high-quality instructional materials, that will give districts the information they need when they go out and adopt.

We also know that's not enough, just writing the reviews. We need to help ensure the reviews are used at the right time as districts adopt and make selections. So we not only write those reviews, we help support some of these IMPD states and districts, not in making recommendations, but utilizing the evidence in those reviews to make smart adoption decisions.

With this, we hope that will improve instructional materials, and thus far we have evidence that it does, because since EdReports launched, over 30 publishers have changed 67 instructional series, moving them closer, or to, standards alignment as a result of the reviews, as well as their own desires. And ultimately, we hope that will lead towards much of the research you already went through, Julia, in terms of having positive impacts for kids.

So when you see standards alignment, how do we do this work? Carefully, over a long period of time, while calibrated with educators. So the first thing we do is we create tools: There are rubrics and very detailed evidence guides around these criteria that we hope are helpful to the field in and of themselves, but we use to conduct the reviews. These are based on seminal research around standards themselves and the kinds of teaching and instructional practice that's necessary using these materials to put this information together.

And we do this with educators—educators who are using these materials. We ultimately go out and recruit and train these reviewers. Over time, we have recruited and utilized more than 700 educators across our seven years conducting these reviews. These teams work usually over four to six months, spending at least five to ten hours per week, utilizing those evidence guides and that rubric, documenting evidence. They get together weekly to discuss that evidence, calibrate, and ultimately use some of the scoring criteria to ultimately put together the reviews.

Our goal is always to work closely with publishers. We know how hard they work to create their materials, and we want to honor them by writing a high-quality, accurate review. So when we work with the publisher to procure and purchase those materials, then we have them orient our reviewers. We asked them questions throughout, and when we complete a review, we go through a rigorous errors and omissions process because our educators are teaching every day, many of them. And ultimately, we want to make sure we capture all of the elements of the materials accurately before we post our reviews and also have publishers put together their publisher response.

We release them, and we feel like if districts are making decisions based on educator-led reviews on these criteria with a publisher's response and having conversations about the content of these materials, we're in a good place.

So I hope, again, you've been to the website and not just looked at the scoring and the color that may go along with a set of reviews; that you've really dug deeply into the evidence where our educators are quite specific in terms of talking about the evidence that they see. It really starts to unpack what are in these materials—what kids are reading, the kinds of pedagogical approaches that are encouraged. And when we talk about standards alignment and the key aspects of those tools in math—and these are our criteria for gateways one and two, we also have a gateway three on usability where we're looking at assessments, teacher supports, the information that helps supports teachers utilize the materials with different learners. But in terms of the standards alignment in math, in gateway one, we're really looking at focus and coherence. Is it the right math, in the right doses, at the right time for kids? And in our second gateway, we look at rigor, all the aspects of rigor, as well as the connection between the content and the mathematical practices. As we go into English language arts, we're really in our first gateway, looking at what kids are reading, are they quality materials at grade level? Do they increase in complexity? Are they the right balance between fiction and nonfiction? And do kids really need to read these texts in order to complete the paths and answer the questions? They're really grounded in evidence. Our second gateway makes sure that kids are building knowledge based on what they read. Collectively, these come together to ultimately define alignment for us around college and career ready standards.

Sy Doan

All right. Hi, everyone. And so I'm excited to talk about the intersection of the two things that Eric and Julia just mentioned, and so specifically is, what happens when we take the EdReport's ratings of curriculum materials and apply them to the American Instructional Resources Survey or AIRS? And so what does that say about the percentage of teachers across the nation who are using standards-aligned material and how that might differ across subjects and grades, states, and years?

All right. So what we have here are the percentages of teachers broken out by grade band and subject. And so on the left hand side, you'll see mathematics broken out by grade band. On the right hand side, you'll see ELA broken out by grade band, again. The percentage of teachers who are using standards-aligned materials—and so to break down exactly how we came to these numbers and what they mean. And so if you're familiar with our reporting within the context of the AIRS, the modal teacher uses more than one curriculum material, or indicates that they use more than one curriculum material, regularly, which we define as once a week or more. And so what we do here to try to get to a single rating is that we generate what we sometimes call a best-case rating. And so across all of your instructional materials, if you said that you regularly used at least one material that was fully aligned, according to EdReports, you're in that green bar on the bottom. And so if you look at elementary mathematics, which is the bar all the way to the left, that indicates that 48 percent of elementary math teachers in 2020-2021 indicated that they used at least one fully-aligned material. And so if we go one bar up to the orange bars, those are teachers that said that they used at least one partially-aligned material, but no fully-aligned materials. In the red bar, those are teachers that indicated that they used at least one not-aligned material, but none of the materials that were fully aligned or partially aligned. In the light gray bars, those are teachers that indicated they only use curriculum materials that were unrated by EdReports. And so those are a combination between titles that EdReports simply hasn't rated yet, but most frequently, the teachers in that light gray bar—these are teachers indicating that they're primarily relying on materials that they created themselves, or that their schools or district created. And then at the very top, you see the tiny gray sliver there. Those are teachers that indicated that they used no curriculum at all. And so as we're sort of comparing across the subjects and the grades, one of the key things that jumps out is that a higher percentage of teachers within mathematics report using a standards-aligned material relative to ELA, with much of that difference being driven by the relatively high rates of standards-aligned material used within elementary and middle school mathematics compared to teachers in the other grade bands and subjects.

And so one interesting thing is that, as I mentioned before, we use kind of a best-case scenario definition of use, but if we look at use in different ways, we get slightly different answers in terms of what percentage of teachers are actually applying standards-aligned materials to the classroom. And so, right here again, we see a breakout by subject and grade bands. But here we have three different different definitions of use. And so in that navy blue bar, that's the definition that we used before. And so these are among materials that teachers say they use regularly. In that middle yellow bar, these are materials that teachers said were required or recommended by their school or district, and teachers could indicate if a material was required or recommended by their school or district, but they simply opted not to use it. And so not all of the materials in the blue and yellow bars are the same. And then at the right hand side of things and that gray bar, these are the materials that teachers indicated that they use for 50 percent or more of their instructional time. And so we're trying to get to, what are teachers main materials? And so we think that this slide tells kind of a very interesting story. So if you look at what materials teachers are saying that their schools or districts are being recommended to them, the percentage of standards-aligned use is slightly higher there. And then once you go down to the materials that teachers say they're using regularly, which is once a week or more, it's slightly lower. And then when we filter down again to the materials that teachers say they're using with 50 percent or more of their instructional time, that's where we see the lowest rates of standards-aligned instructional material used. And so this is sort of an interesting illustration of the different pipelines and the different leaks in a pipeline that can occur when trying to apply standards-aligned material used within the classroom.

What you see in the slide here are each of these bars indicate the percentage of teachers that indicate regularly using fully-aligned math curriculum materials. And so what you see here is that it's broken out both by state and by year. And so we have three years of bars here for the three years that we've done the American Instructional Resources Survey. With the national numbers all the way to the left, and then state-specific numbers for the IMPD states in the state bars to this right. And so if you look nationally, in 2019 34 percent of math teachers indicated that they used a fully-aligned math curriculum material, and that jumped to 46 percent in 2020 and then 42 percent in 2021. And so one important note here to note about the 2020 numbers is that, as I'm sure everyone here remembers, that was sort of the onset of closures due to COVID-19. So when we administered the survey, we asked teachers specifically to indicate the materials they were using prior to their schools going remote or closing down due to COVID-19.

And so a couple of interesting patterns emerge here. Is that relative to 2019, nationally and in many states, we do see an increase in the percentage of teachers indicating the use of fully-aligned math curriculum material, with steady increases in certain states such as Delaware, Nebraska, Rhode Island, and Wisconsin. And then Louisiana, which had a very high rate of use in 83 percent and then dropping slightly in 2020 and 2021, but still remaining among the top states in terms of the percentages of teachers using a fully-aligned math material.

And here are just the analogous numbers for ELA rather than math, and so as we mentioned before, the rates of teachers using a fully-aligned material in ELA is lower than it is in math. But here we see kind of more consistent patterns of growth across many of the IMPD states. So again, Delaware in 2019—11 percent of their teachers indicated using a fully-aligned ELA curriculum materials. And according to the data that we have in 2021, this number jumped to 33 percent. And so it is a great indication of a place or state where we saw a lot of growth. And so similar growth in Mississippi, Nebraska, Rhode Island, Tennessee, and Wisconsin.

And so one of the big questions is, was there a difference between states in the IMPD network and states not in the IMPD network? And so what we did here in this table is that we took a look at what we call the IMPD states that have been a part of the network since the first year we administered the AIRS. And so that's excluding Arkansas, Kentucky, Texas, and Ohio, which are new states to the IMPD network in 2020-2021. And so what we see when we do this very high level of comparison is that across all years and subjects, with the exception of 2019-2020 in ELA, teachers in IMPD network states were significantly more likely to use a standards-aligned material than teachers not in the IMPD network.

And so here we're going to have a poll to ask—and so this is sort of the big question to us and on our end of things we're still trying to piece through, what is exactly driving the rise in standards-aligned material that we see? So we're very interested in your perspective on it, and then to think through some of the possible reasons why, and then have a sense. And keep that in mind as we're kind of discussing these results and hopefully some intriguing Q&A at the end of this session.

And survey says... So the most common result we have here is that states are providing more guidance and incentives for use of standards-aligned materials. And then that, I think, is one of the most exciting things and a lot of the conversations we've had with different state departments of education, there have been a ton of efforts just to provide more very usable information and detailed guidance on how to use these materials. And so that's great. And there's a greater and greater press for students to master math and ELA standards, which is in second. And then tied were—more standards-aligned materials are available and more information is available about what is standards aligned. And so kind of touching on all the great work that Eric and his team at EdReports, as well as many other state departments of education and many other agencies, just making that information about what is standards aligned more readily available.

All right. As I mentioned, 2020-2021 was the first year in which the AIRS was administered specifically within the instructional context of COVID-19 and what that meant in terms of being in-person, being hybrid, or being remote, in terms of instructional mode. And so we have some results from the survey that we wanted to talk about to kind of give more context to what instruction and instructional material use looked like during this year. And then we'll have EdReports come in and describe how they're rating the different technology and accessibility features of specific materials at the end.

And so one of the questions we asked was, on average, how many hours were you devoting to instructional planning? And so we wanted to see how that might have differed across different grade bands, as well as teacher-reported instructional mode. And so on the left hand side, we see these are teachers that were fully remote. In the middle is a hybrid instructional mode and on the right is fully in-person. And so one of the big differences that we can see here, the teachers in fully remote and hybrid learning environments on average spent more hours in terms of their instructional planning than teachers and fully in-person settings, with the bar for elementary teachers and fully remote being the highest. And so those teachers on average spent roughly 21 hours per week doing instructional planning.

Another question we asked was, we asked teachers, given this point in the year, if you compare yourself to how much content you covered in the prior academic year, how on pace were you? Or approximately what percentage of that content have you covered? And so what you see in these bars here are the percentages of teachers who responded that they covered nearly all or all of the content in 2020-2021 relative to 2019-2020. So to put it a different way, these are teachers that were, quote unquote on pace with what they had covered in the prior academic year. And so here you can see another big difference between fully in-person teachers and teachers who were not fully in-person, where much larger percentages of teachers in fully in-person settings were more likely to indicate that they were, quote unquote on pace with content coverage relative to teachers in fully remote or hybrid settings.

And so we have another question here. And so we saw that, particularly for remote teachers, lower percentages of those teachers indicated that they were on pace relative to what they had covered in the prior year. And so what do you think is one of the biggest reasons teachers gave for lower curriculum coverage in 2020-2021?

The top answer we have here is that, with 48 percent of you, indicating that students have had less classroom instructional time than last year; followed by curriculum content and use last year was not visually accessible; and in third place, students needed more remediation activities than last year. And so fantastically, we have the answer based on teachers in the next slide. And so we asked teachers to indicate and to rank in terms of what was most important. And so they could say that this was the first most important, second most important, third most important. Those very same reasons that we asked you in the poll itself. And so in terms of teachers most likely to indicate what was most important, students having less instructional time was the most frequent choice; followed by students need more remediation; and in third place was the content not being digitally available. And so one of the interesting things is that "other," particularly in terms of being third most important, was also a frequent choice. And then something that teachers indicated a lot in "other" was that students were not engaged or students were not doing homework and the assignments in order to advance through the curriculum.

And so with that, I'm going to switch back to Eric who's going to talk a little bit more about EdReports and what they do with key technology information.

Eric Hirsch

You bet. Thank you, Sy. And we have some information in our gateway three on usability about some of the aspects of engagement supports for teachers and other scaffolds, and assistance that will help kids who are behind, and to really personalize instruction to all students. But once the pandemic began, we got a lot of questions about the digital nature and whether or not some of the standards-aligned material we had reviewed would work in virtual, hybrid, or in-person settings. So one of the things we knew we needed to do to help around our mission and to inform the field as they made choices about what materials they would select, was to get better information about technology. We always had an unscored section around technology because technology varied—when we started, we reviewed some things that were purely print, and we certainly review things that are wholly digital. So we went and asked publishers themselves to answer a host of high-level questions about its applicability in the various settings—hybrid, remote, and in-person—as well as share things such as which LMSs did these materials align with, and a host of very specific technical questions to try to really unpack and answer whether or not these core materials may be digitally appropriate across those settings.

So I want to make clear this is not evaluative. We did not score. We would go back to publishers and clarify our definitions and sometimes ask for additional information. But this is all publisher provided information. You can see it in this screenshot about where you can find this additional information or which reports will have that PDF at the top that has the technology evidence.

And so if you go next slide, a few things. You can see some of the details, how we ask across settings and the like, and just a few high-level pieces across the roughly 150 grade level reports that were enhanced with this technology information. First off, I would not presume that your core materials may not work across various settings. According to the publishers and the information they provided—and it's up for you to assess that information—that most of the standards-aligned materials claim to have technology that facilitates student learning in remote and hybrid environments. How well they work in those settings—for example, students work within the materials that save versus what's done on an LMS—that varied a lot. But many of the products had native digital features. But we saw that what publishers deemed as working remotely, and what had remote learning capabilities, varied a lot. So I encourage any of you who are looking at materials, want to prepare for a pandemic that hopefully ends, but also, there may be a variety of reasons why students may need to be virtual or in hybrid settings across the school year, that we hope this information is helpful, but really dig in and review the details. We say the same in gateway one and two around some of the alignment criteria, and really unpacking what these materials are designed to do. And that is absolutely true around technology.

Julia Kaufman

Thanks, Eric. I'm going to talk a little bit about the factors that we are learning encourage curriculum use, including use of standards-aligned curriculum. This is just some of what we've learned through our American Instructional Resources Survey reporting. We've learned that being in an IMPD state is related to teachers saying they're using more standards-aligned curriculum. That may not be shocking, but we've even found that to be the case when we take into account a lot of other variables. So it might be that an IMPD state has more of a certain kind of school and that certain kind of school uses more standards-aligned curriculum. In this case, we did try to take into account the income level of the school, the ethnicity of the students, all kinds of variables, and even then being in an IMPD state was predictive of use of more standards-aligned material. But in our reporting, one important reminder is that in our reporting, we did find that even when IMPD states were first joining the network, the use of standards-aligned materials in those states was higher. So those states may have joined the network because they're really focused on use of standards-aligned materials.

Some other factors that we found in our reporting and our analysis of our data to be really important and tied to more use of standards-aligned curriculum—and by more use, we mean reporting that you use it regularly, but also that you don't modify, you know, more than half of your lesson, for example, because we would define that as more use of the curriculum, right? So we found that when teachers report they're getting professional development that is focused on curriculum, they report more use of standards-aligned curriculum. We also found that when they report that their principal gives them feedback and evaluation that's focused on curriculum use, that's related to more use of standards-aligned curriculum. And it's not just standards-aligned curriculum, it's use of any curriculum, right? So that's sort of a lesson to any school system is to think about what are the messages that we're giving them through professional development, through principal feedback, through principal evaluation? And it turns out that if those messages are curriculum specific, that will likely encourage more use of the curriculum that your school system has required or recommended.

Now I'm going to pass it back to Eric to talk a little bit more about those IMPD states.

Eric Hirsch

Thanks so much, Julia, and I couldn't agree more. In fact, research backs up if we're going to change instructional practice, making sure the use of curriculum is a priority as we have conversations with individual teachers around evaluation and continuous growth. And as we think about professional learning that implementing a curriculum to actually get to what teachers do in the classroom is critical.

But I do want to focus on that first piece of the IMPD states having an impact on the use of standards-aligned material have all followed with these states prioritizing high-quality instructional materials—creating a systematic plan within the department and with other partners and colleagues to really think about ways in which they can be supportive of districts in selecting and ultimately implementing high-quality instructional materials, as well as teacher preparation institutions in getting teachers ready to become curriculum literate in terms of using these materials.

Each of them have taken on some strategies that seem to fall into three buckets of work. We have some examples of these, and I really want to shout out. I see so many state education agencies, CAO, friends and colleagues from some of these states. So I chose a few of you as exemplars, and there are so many exemplars not listed and many folks in the webinar who can, I hope, jump into the Q&A and share more information around some of these.

This first arena in which states work to really enhance the use of HQIM is first off, just gathering information and being transparent and clear about what is in use in the first place. It continues to amaze how little we know across many states about what districts have actually selected, let alone the challenges of understanding whether it's actually utilized in classrooms. So several of these states said, How can we, first of, understand what's in use and make it known? And shout out in particular to Massachusetts and Curate, and Nebraska and their Materials Matter websites, where they've come up with heatmaps. So it's not just information for the state, it's information for other districts, so they know what their neighbors and peers are using, so they may be able to reach out and better understand. So the first part of signaling is understanding the landscape in the first place.

The next is helping folks understand what really is high quality, what is standards aligned. Many of them have utilized our definition at EdReports, others through stated options. And there are 18 states that adopt, ten advisory, eight mandatory. And many of them have really worked on that signal. And they've bolstered adoption criteria, and they've really become quite rigorous around what needs to be in these materials in terms of content to really make sure we're conveying standards. And real shout out to Mississippi, New Mexico, and Rhode Island, that have put out lists that have worked hard to recruit and train educators. And we at EdReports know how hard that work is and how detailed. And they've created lists that really help inform districts not just about the choices themselves, but the evidence and the information about those choices.

Once we know what HQIM is, these states have worked hard to incentivize their use, either through making state procurement for things on the adoption list simpler and more streamlined, and in many cases, providing access to high-quality instructional materials, particularly in the pandemic, with the ESSER funds for free. So shout out to Nebraska even though you're not on the list, Corey, but I know the work you've done. And just wanted to call attention to Texas and the Texas Home Learning website where they've made HQIM materials available to all districts in the state.

Finally, not only signaling and incentivizing its use, but getting to the professional development aspects that you put in the research, making sure there are supports. Making sure at the state level that you're making sure professional learning is curriculum specific, as well as content focused and around specific instructional materials choices. A good example of strong supports just shouting out Tennessee and all the work they did in literacy and they're reading 360 program. And there are so many examples from the other IMPD states. So signaling making sure districts know what are good choices. Incentivizing those choices to help ensure that they are picked up, and then making sure through supports that those materials are used.

Ultimately, what does this mean for moving forward, and some of our work? We hope some of the take away from some of these policies and seeing the rise of standards-aligned material is—there is a lot districts can do to select and demand high-quality instructional material. Our reviews have shown more and more standards-aligned materials are becoming available and that districts should make sure standards alignment is a key criteria as they work locally. They can put that in their RFPs, they can work on local criteria, and they should really work hard to engage educators who will ultimately be using these materials in the selection. And then they understand why. But it's not just teachers, it's engaging the community, caregivers, families, other educators to make sure everyone understands why materials may be high quality and why they should be utilized in instruction.

I want to be clear we talked here about standards alignment, but we do not think alignment is the pure definition of high quality. Alignment is necessary, but it's not sufficient in defining HQIM. So again, encourage you to dig into our reviews and look at more and more evidence about things like, whether the materials are culturally centered, whether they create engaging, authentic, meaningful experiences for kids, aspects of usability that I had brought up before around the assessments, the teacher supports and students support, and the like. And finally, we know, given the pandemic and given some of the focus of our current context, that technology has become very important. But oftentimes, curriculum has been thought about without the folks who do technology in a district, and technology has been thought of without the content and curriculum. Understanding the capabilities of the materials around support across the different hybrid, remote, or in-person settings is going to be critical. And while tech supports are important, we want to remind you this whole webinar is about standards alignment and the content is crucial.

Julia Kaufman

And I just wanted to add a few implications of our data that we've been thinking about a lot. Over time, we've actually been looking at use of instructional materials in many states even before the American Instructional Resources Survey, and all of our evidence, including some studies we've done in Louisiana, suggest to us that states can play a major role in incentivizing use of materials. We see standards-aligned material use growing in states that are focusing on that. And we believe they could be a model for other states.

But one important point is that incentivizing use of standards-aligned materials is only a first step to ensuring high-quality instruction. I saw a question in the audience about the point that materials is a means to getting to standards-aligned instruction. And while we've been trying to figure out the extent to which standards-aligned material use is aligned with instruction, that really likely depends on so many factors, including how much you're using those materials, but also how much you really understand those materials and how they're teaching the standards and the support you get as a teacher to teach to those materials. And of course, as Eric mentioned, there are other dimensions of quality that many of you are likely thinking about as you're watching this presentation, including the culturally-relevant alignment of materials, how much they support English language learners, how much they support special education students, et cetera, et cetera.

So we can only do so much through the work that EdReports reviews is doing and what we're trying to do to code materials. But we are thinking about ways to code or take into account the cultural relevance of materials. If we can find more data or more places that are rating lots of materials, we'll be able to do that because we're asking teachers, again, what they use, not whether what they're using is standards aligned. So we can use what they use as a measure for helping us figure out whether they're using things that are, for example, more culturally relevant.

And lastly, states and school systems and researchers should be tracking notable shifts in use of standards-aligned materials. And I guess that's sort of a self-serving implication. But when you do track shifts, we believe that you can pinpoint the policies that might have led to those shifts, and you can even move further to think about whether those shifts are resulting in learning improvements. We are thinking about that last piece, which of course, is critical to really understanding how standards-aligned materials is leading to student learning.

OK, now I think we'd like to pause for a little while to try to answer some of the questions that we have not been able to answer as of yet. If we've answered your question and that wasn't enough, please post, you know, another question, a follow-up question, so we can try to answer that. I know Sy and Mark and Eric have been busy responding to all of your questions, and it's been great to see so much interest and so many questions. But I'm going to pause for a moment and see if there's anything that Mark or others think we should kind of lift up and talk about a little bit more, and whether there are other unanswered questions.

Mark LaVenia

Sure, I'll go first. There were so many great comments and questions, it's difficult to pull a few. One, I think this would be directed to Eric specifically—as academic standards seem to be evolving from skill standards, common core, ELA, and math, to process standards like NGSS, C4, etc., how is that impacting your review of curriculum terms? What challenges are presented when reviewing curriculum materials that attempt to support the "how" of learning, as well as the "why"?

Eric Hirsch

Yeah. And Mark, I welcome you to join me here as you sit with me at EdReports with our colleagues working with these excellent educators across the country.

Even within math and ELA there are a lot of examples around the "how" that we try to look at whether they're captured in instructional materials. For example, the mathematical practices and their connection to content are a lot around how we're teaching that content, and we are certainly already reviewing in science for alignment with NGSS. So we are looking at things like how are phenomena used and sense making? And are those phenomena really driving learning? The answer is it's really tricky and challenging, and we do our best. We know our best report is yet to be written, and getting at things in the materials that may provide information and guidance through teacher instruction, through additional information and teaching strategies, are the way we try to bridge and look at the materials themselves and whether or not they're giving teachers what they need to first, understand the "how" of what it would mean to teach towards these standards, as well, again, as the specific guidance and instructional strategies within. Mark, do you have anything to add? You sit in the EdReports virtual chairs alongside me.

Mark LaVenia

I think you nailed it. I don't have anything more to add to that.

This one is a perennial discussion, and anyone on this call could speak to this, I'm sure: I found that more teachers view themselves as experts at teaching reading or ELA, regardless of their background. This leads to a notion that they don't need to follow an aligned curriculum and should be able to teach what they choose, as they choose. Conversely, many teachers do not view themselves as math experts and are more likely to actually use the math curriculum. So we could have a whole other hour on this topic, probably. But anyone want to jump in? Please share your thoughts.

Julia Kaufman

I mean, I'll just share a couple of thoughts. One is that teachers have for a long time, I think, regarded themselves as creators and developers of curriculum. I can tell you when I got my masters in teaching and in teaching secondary English, a lot of our work revolved around creating our own curriculum units and using them in the classroom. And I regarded that kind of as part of my identity as a teacher. I think that it does make intuitive sense, though, to think about sets of curriculum that we can draw on, modify as needed. Because if every teacher in the United States is reinventing the wheel, students are going to be learning very different things. And so I think modifying curriculum, adding to it, making sure that you're supplementing when students need it, all that is so important for teachers to do. But I think we need a new kind of skill set to be really focused on in teacher preparation programs and training. And that is how to use curriculum well and productively and thoughtfully. That doesn't mean that you're just, you know, automatically using what you've been given exactly as written. I don't think that always makes sense for all students. But to be able to draw on content that's similar across all of your students, across a school, across a district, across the state, that makes a lot of sense to me. And so I think we are sort of starting to move in that direction. And I think we also need teacher preparation programs to support that by helping teachers become really thoughtful users of curriculum.

Eric Hirsch

Yeah, I couldn't agree more, Julia. We oftentimes talk about instructional materials versus curriculum because there are perceptions sometimes baked in early in teacher preparation all the way through, that the materials are meant as a script. That they're meant to follow in lockstep day to day. And we think about core curriculum, and the scope and sequence, and the content it provides, as a support for teachers, as a skeletal system, as something to work off of and build off of. And the hard work of teachers is using that curriculum with integrity and skillfully. And so I do think, as per the question, there is a degree of comfort sometimes doing that around ELA, versus mathematics. And I think that's not just true in the content areas of what we've seen in your awesome data from RAND, but in our own data and our own conversations with the educators, is there's also a lot of variation across grade span. So the use of instructional materials both in ELA and math is less in high school in terms of using it 50 percent of time or more as per the RAND definitions and using it regularly, than at the middle school and elementary school, as well. And some of that could be based on content and familiarity and comfort. But it also can be around the design of the materials and how easy they are to supplement and adapt as well across the content areas.

Mark LaVenia

Thank you. We still have few more minutes. One just came in and I think there's a lot that could be said from both EdReports and RAND. And that's, some questions around alignment between HQIM and assessments. So in terms of, you know, what kind of work are we doing, in terms of EdReports doing, in terms of pushing in the area of that connection? And in terms of RAND, wonder if the AIRS data would help illuminate these connections between curriculum and assessments, and maybe you've already done that? So if anyone wants to speak to that aspect?

Eric Hirsch

What a fantastically easy question. I'm happy to go first. We do look deeply at the assessments, both in gateway one in our standards alignment aspect and in particular in gateway three on all assessments—more of the culminating unit assessments are in the gateway one on standards alignment—about what's in the curriculum in the first place. Because we know as we think about the instructional core, it's not just the content and the scope and sequence, but getting accurate information on what kids are learning that ultimately will come together and actually answer some of the things Julia was talking about, about when you may need more supports with certain students and the like. So we're already reviewing the assessments that are part of the curriculum purchase. We have been investigating, looking at interim assessments, and some are designed specifically around a curriculum so they can not only provide information about a student's knowledge, but perhaps how to better utilize those curricular materials and where to go within the scope and sequence and lessons. And then some are standalone interim assessment products. We think those are really critical given the information they provide, and we hope to talk to you at our next RAND webinar about assessment products.

Sy Doan

Just one comment to add in there about state assessments is that there is an item in the survey this year that could point to this, but unfortunately, we don't have the data rung up yet. Is that we do ask teachers to kind of rate the adequacy of the materials with regard to the state assessments in their state. And so that would be a very interesting state by state comparison to see whether or not there are teachers at some states feel that the materials that they're using are more or less adequate with respect to the assessments being administered in that state.

Julia Kaufman

I'm sorry that we haven't gotten to all the good questions that you all have. We do have contact information I'll show in a few minutes so you can reach out to us via email to ask some of these great questions. I just wanted to briefly go over some of the things to expect from our American Instructional Resources Survey. Our report on use of standards-aligned materials is out today—yay!—at the same time as the webinar, so you can take a look at some of the data from the webinar in that report. And we also have a technical document that accompanies that report that goes into some detail on our methods. We also have asked a lot of questions in our surveys about use of anti-bias materials across K-12 teachers in our sample, and we have a report coming out on that in December. And then in February, we're going to get more deeply into what's happening across states, and particularly the IMPD network states, in terms of both use of material, but also the professional learning supports for their use. We'll be doing another AIRS survey in April 2022, and we're really excited to be thinking about what next, what we should be asking about instructional materials that we haven't asked about yet.

There's an email on here for Aaron Lang, who works with me, and if you want to be included on the email list for any of those reports, just let us know. We also have a lot of other American Educator Panel projects, including the American School District Panel and the state of the teacher and principal surveys to be fielded this winter. So just be on the lookout for all those results.

Lastly, I just want to mention Bento from our friends at Kitamba. It's a free to use data visualization tool that lets education leaders easily view, explore and understand American Educator Panel survey data. It currently has the AIRS 2019 and 2020 data in it, and the AIRS 2021 data will be available in early November, and you can take a look at that data by state. So really dig into it.

Eric, did you want to add anything else about what EdReports has going on right now?

Eric Hirsch

Lots and lots of reviews. We just published two math reviews and an ELA review, and more to come. We are working with a group of educators to expand into high school science, so we will be in K-12 ELA, math, and science. And I assure y'all there's been some wonderful questions in the chat about assessment, about social studies, about culturally centered, and we continue to learn and try to reflect on our own criteria, and are excited to keep providing as much information as we can to the field to help empower local districts to make great selection decisions.

Julia Kaufman

Well, thanks so much. It's the top of the hour, so we'll bid you farewell. I hope you have a good rest of your day and we really thank you for joining us.