Introducing a Toolkit for Improving Instructional System Coherence

Published Dec 13, 2023

by Julia H. Kaufman, Elaine Lin Wang

Broadly defined, coherence in instructional systems for kindergarten through grade 12 is the circumstance in which all instruction-related messaging and supports—for example, via curriculum materials and professional learning—provide teachers and other instructional staff with clear and mutually reinforcing messages about what to teach and how to teach it. District and school leaders have the primary responsibility for developing coherence by providing those materials, messaging, and supports.

RAND researchers developed a toolkit, along with a workbook and slide deck, to help district leaders, school leaders, teachers, and other instructional staff investigate and reflect on the extent to which coherence is present in their instructional systems, with the goal of improving that coherence.

RAND Education and Labor hosted a webinar in which researchers introduce the toolkit and how to get started using it. Alongside this introduction to the toolkit, a K–12 public school district administrator and educator reflect on the importance of improving coherence and challenges that users of the toolkit might encounter in doing that work.

Transcript

Julia Kaufman

So I'll introduce my fellow panelists momentarily. Elaine Wang has co-led this work with me. In addition, I'd like to acknowledge the others who made substantive contributions to this toolkit. My colleagues, Kate Kennedy, John Schweig and Kate G. Will. Throughout, if you do have questions, please enter them into the Q&A. We'll do our best to answer them on the spot if we can, or save them to the end. But please do ask questions. There are no dumb questions here. We are happy to save them for the end. But again, we'll try to answer them if we can.

Julia Kaufman

So to access the Coherent Instructional Systems toolkit, it is online right now at the URL that you see there. It includes the user guide there as well as an accompanying workbook and an accompanying slide deck. We created those things so that you can use them during team meetings to work through the toolkit steps. And Elaine just shared the URL there, which is great. So I am so thrilled to have these other presenters here. So I'll introduce myself first. I've been undertaking research related to coherent instructional systems for many years, particularly interested in how curriculum materials align with standards and what states and districts can do to support adoption and use of high-quality standards-aligned materials. I've done particularly a lot of work to study education policies and systems in Louisiana, and I've done considerable research to understand the extent to which surveys can give us accurate information about what is happening in schools and classrooms.

Julia Kaufman

Elaine Wang, who leads this work with me as a policy researcher at the RAND Corporation. Her primary research interests concern K-12 instruction, especially with a focus on literacy and English language arts. She also researches the development of school leaders and specializes in mixed methods to examine relationships between instruction and student learning outcomes.

Julia Kaufman

Elaine Wang, who leads this work with me as a policy researcher at the RAND Corporation. Her primary research interests concern K-12 instruction, especially with a focus on literacy and English language arts. She also researches the development of school leaders and specializes in mixed methods to examine relationships between instruction and student learning outcomes.

Julia Kaufman

I am so thrilled again and honored to have our two panelists today. First, let me introduce Dana Talley. She currently serves as the chief academic officer for the Lincoln Parish School District. Prior to returning to Lincoln Parish, she worked for the Louisiana Department of Education for 15 years, which is when I first met her serving as a network network leader, a deputy network leader, and a network coach, and a distinguished educator. Probably not all at once. Her primary role as a network leader was supporting 34 parish superintendents, district leaders, and principals in the implementation of high-quality curriculum materials and allied professional learning. And then last but not least, Emily Howell serves as the English language arts facilitator for Lincoln Parish Schools in Ruston, Louisiana, overseeing curriculum and instruction for ELA in grades 3 through 12 and also teaches in the classroom part time. Over the past decade, Emily has developed and presented ELA content for the Arkansas Advanced Initiative for Math and Science Applied Practice Propel, laying the foundation in the National Math and Science Initiative, as well as the Louisiana Department of Education. She's developed curriculum for both learner civilian and the Louisiana Department of Education, and she was the 2012 Louisiana High School Teacher of the Year. Again, I'm thrilled to have them all here to join in this discussion. So today, we're going to talk a little bit about instructional coherence, and then we'll give you an overview of the toolkit and go a little bit back and forth on that to discuss with our panelists how various steps might be implemented in the classroom and what to think about. So for now, I'm going to turn it over to Elaine, who is going to give you an overview of our conceptualization of coherence.

Elaine Wang

Great. Thanks, Julia. I'll lay the foundation and we can go to the next slide. So let's define coherence in our conceptualization. Coherence describes the extent to which components of an instructional system convey a clear and consistent vision and direction for educators and students so that teachers have a good sense of what to teach, how to teach in a way that's aligned. So what do we mean by the components? In our research, we have been focusing on seven specific instructional system components, and they are state standards. For example, the state standards for K-12 or math instruction curriculum materials presumably aligned with standards, formal school or district offered professional development. And then another aspect of professional development that we're calling teacher collaboration guidance, and that's guidance provided around what to focus on or how to use peer collaboration time. Then there's the processes and criteria with respect to teacher evaluation, and then there's interim assessments or benchmark assessments. You can think of it as that. And then there's the state mandated standardized test for the subject, if applicable. So these are the seven components that we focus on in our study and our research around coherent instructional systems. I'll try to make it come alive for you a little bit. We'll take you through two brief vignettes that help illustrate a less coherent and more coherent instructional system, and you might recognize features. Of your school or district through some of these vignettes. We'll start with Mr. Garcia. He's a sixth-grade teacher at Maple Hills Middle School, a pseudonym, and he teaches students from a variety of backgrounds. Let's say that he's required to use EL education as the curriculum for instruction in his sixth-grade class. He uses it regularly with fidelity for the most part, but he adjusts the curriculum material to support student needs. But essentially, essentially, as Julia clicks through, you can see that he's using curriculum material that's aligned with standards. He uses district-developed early benchmark assessments to set targets for students and student achievement. But, this, these assessments are not aligned with EL education. So you see the red line there. We're using that to represent kind of lack of alignment or lack of coherence between the two components. And this is because the assessment was developed years ago and it aligned with the prior curriculum, not the current Yale program curriculum. So there's a disconnect. Still, he's, and because of this, he's unsure about how to use benchmark assessment results to drive this instruction and to use the curriculum. So he ends up picking and choosing activities that he feels matches the upcoming benchmark or the upcoming STAR assessment that the district requires them to use. As for professional development, Mr. Garcia receives PD that introduced EL education when it was first adopted, But this year the school wide PD is not focused on EL education or even ELA content. Instead, it's emphasizing social, emotional learning. And Mr. Garcia isn't sure how he should or should not use EL education curriculum to address that focus. So we can put that on the diagram as well. There's a disconnect. And then lastly, Mr. Garcia has weekly collaboration time with other sixth-grade teachers. But the district and the school have not provided much guidance on what will be on the agenda for those meetings or what topics teachers are to discuss. So they end up discussing assessment results sometimes, or sometimes they use the time to coordinate logistics, like which students to take to pull-out sessions or how to communicate to parents. And sometimes the peer collaboration time is not strictly used to discuss instruction or curriculum. And so, in sum, we interpret that the system Mr. Garcia teaches in is rather incoherent. It sends different signals, sometimes mixed signals to Mr. Garcia about what and how to teach. Let's click through and meet Ms. Martin. let's say she's also a sixth-grade English teacher, teaching students from a variety of backgrounds, and she also uses the EL education as their curriculum. And she also strives to use that with fidelity in her system to support teachers to use this curriculum, the district adjusted their interim assessment that they had been using that was in place before EL, but now they ensure that the interim assessment has sections explicitly aligned with EL curriculum in the scope and the pacing that they've worked out. So the district emphasizes EL education, performance tasks, and the end-of-unit assessments that come with EL education, and they use it as important indicators of student progress. The district also provides discussion protocols for teachers to use during teacher collaboration time, and these protocols help to drive teachers toward examining student work associated with the assessments. The district indicated that teachers should use the collaboration time to identify common student errors or misconceptions, or brainstorm ways to reteach, reinforce skills, and which EL education lessons to pull on to do that with. So then teacher collaboration time is connected with interim assessment and with a curriculum at the least. Like Mr. Garcia's scenario. Ms. Martin's district PD also focuses on social, emotional learning. She was initially worried that SEL would be tangential to the focus on implementing EL education the curriculum. But in PD sessions, teachers are grouped together by grade-level subject area, and they discuss how SEL can be compatible with L education, with the academic curriculum, and how they can be mutually supportive and integrated. When Ms. Martin meets with a principal in advance of her formal teaching evaluation observations, they review the evaluation criteria together. They discuss, the principal discusses with her feedback on how she uses Yale education, and they also review recent performance on students end-of-unit assessment. And they look at data so that that can drive her instruction. So in this way, the evaluation process, the teaching evaluation process is also linked to curriculum into interim assessments. We consider Ms. Martin's instructional system then to be rather coherent as exhibited by these screen lines. So what do we care? We do know from some of her research and some of her reports that incoherence and coherence affects teachers. In interviews, teachers have told us that system coherence increases their confidence with their work, increases their ability to be flexible and creative and take risks and coherence, helps them and helps them see students as being successful. On the flip side, we've had teachers tell us that incoherence increases their frustration levels, makes them feel overwhelmed and allows them to feel confused, discouraged, and overwhelmed. And misalignment and messages also contribute to a lack of trust between administrators and teachers. We've had one teacher tell us that they sense different messaging from different components of their system and leading him or her to say maybe the administration is trying to make us fail. So this in part forms an impetus for our work with toolkit and why we think schools and districts should examine and strive to achieve instructional system coherence. I've just given you some quotes and perspectives from teachers that participated in our prior research, but since we have Emily, a very well-seasoned teacher here, I wanted to invite Emily to give some remarks and have her comment on how her experiences have fit with what we have described and how coherence or lack of coherence impacts your instruction. Emily. Great.

Emily Howell

Thanks so much. I think we've all been in Mr. Garcia's position at some point, and speaking from a personal standpoint, prior to our system becoming coherent, we really had no Northstar in terms of curriculum. We did not have reinforcing messages about what to teach and how to teach it. And as a result, there was a lot of curricular chaos and confusion among teachers. One of the things that I would see the most, even when we had curriculum, but it wasn't being used with fidelity. I would see low quality curricular materials or teacher-created materials. I would see teachers cutting things, replacing things, really bad pacing issues. And the problem was we we had competing priorities and messages in our system. It wasn't anybody's fault. I think that everybody was working really hard and wanting the best for kids, but we weren't rowing in the same direction. And like I said, that that effect was curricular chaos. And I found that when teachers don't have curriculum, they spend all their time creating, not actually becoming masters of content. So in short, we just did not have the conditions in place for people to be successful.

Julia Kaufman

Thanks so much, Emily. At this point, I'm going to start walking through the toolkit steps. And before I do that, one of the things I just wanted to do is take the pulse of who's in the room so we can think about that as we're going through each of our steps. So we have a little poll for you, if you wouldn't mind, just so we can understand who's participating. So please let us know. Are you a district or school administrator, a K-12 teacher, a researcher or faculty member, a state or federal official or staff staff member at an organization that supports teachers or school leaders or something else that we neglected to include here? So give you a few seconds to complete the poll for us. Great. Get a nice spread. No teachers today, but we are recording this webinar, so hopefully teachers want to will get a chance to view it. So we've got a nice spread here. That's great. So now I'm going to walk you through the toolkit steps, of course, with the user guide that I showed you and the workbooks and a set of slides. There's a lot beyond what I'm going to be talking to you about today, but I'm striving to give you an overview. You need to ask questions and maybe even get started and thinking about it and whether this is the right tool for you. So this part of the webinar will be more of a conversation. After I talk for each step, I'm going to go back to our commentators, our panelists, and ask them if they have any thoughts based on their own work in like Lincoln Parish in Louisiana or elsewhere. So let's just give you an overview. So the toolkit has four steps. One step, determine the focus team and timeline really be getting started. The next step, identifying a North Star and focal instructional system components. I'll talk to you more about that in a minute. The third step, reflect on coherence and then the fourth, create an improvement plan. So in a way, the first three steps are all pre-work for step four, which is really about digging in and engaging in a plan. Each step, as you see there, has some tasks associated with it. So for the first step, you're figuring out what the focus of your work should be. Assembling a team who's ideally excited about undertaking this work and setting up a timeline and having a kickoff meeting for the second step. That's an important one. You're deciding on your North Star. So what is it that you want your system to be coherent with? Is it your materials and academic standards? Is there something about your school or district's mission that you need the team to keep in mind when thinking about coherence? And then what are the key components or levers that you want to make sure are coherent with this North Star? We've already talked about them. A little professional learning is one opportunities for teacher collaboration, interim and summative assessments, evaluation. And beyond that, there may be other components that make sense to include. We'll give you some examples of other components that you may be thinking about in your system. The third step is actually doing the work, making sure that all these components, understanding whether all these components are coherent, agreeing on that as a group, thinking not only about coherence across all of your students, but whether teachers and students serving students with different demographic. Characteristics are also experiencing that coherence in the same way. So like students who are English learners, who have individualized education plans, Black students. So thinking about that step four, there's a lot in this step. This is actually the part where you work to improve everything that you did all that pre-work for. We didn't make this toolkit document a comprehensive primer on how to undertake school improvement efforts, but we highlighted the major steps to include in this part. And then and so those major steps include like identifying areas to improve, brainstorming, creating a plan, sharing that plan with stakeholders, which is a really important part of the process and getting their input and then just stating the work and not forgetting about it. So I'll talk through each of those steps in a little bit more detail. I'll give you an overview of what you'll probably be wanting to think about and then again asking our panelists how they respond to some of these steps. Now, at the very beginning of your work, you or whomever has initiated this work and decided is important is doing a lot of planning. Even before you bring the team together, you have to consider where your focus will be. It could be K-2 math or six through eighth grade ELA or high school social studies. It could be lots of things, but it's important to start relatively small. Improving everything is probably not a recipe for success, and it also provides a lot of different things to talk about. You've never done this toolkit before, so I would recommend considering what grade levels and subject areas could benefit the most. To start and see if this works for you and your school. Your team also needs to include all the roles and critical voices. So for example, if you decide we need to focus on middle school math, you probably want to include someone who has some kind of decision making power in the middle school to buy into this work and understand it and weigh in. So like a principal or a vice principal or someone who focuses on math academics at the middle school level, he'll be able to get the things done. You decide you want to get done. You probably want to include math teachers. If your focus is on middle school math, maybe at sixth grade, seventh grade, and eighth grade. If you have math coaches, you may want to include one of those. So think about your team first and then have a kickoff meeting We provided you with the slides and the toolkit and lots of guidance for running the meetings. It's a moment to make sure everyone is on the same page about what they're doing and as bought in. And one important message to convey is that it's not about improving teaching, but about improving the systems that good messages to teachers about what to teach and how to teach it. So I'm going to just pause here and ask Dana to give her expertise or her thoughts on what are the most important considerations for this step based on all her work.

Dana Talley

Thank you, Julia. I have to say, just upfront, this was really, really hard for me. And when I first started looking through this toolkit and was talking to Julia, I really kind of wanted to push back here because I want to fix everything yesterday. And my thought is, if we fix one subject in grade band a year, it's going to take us 20 years to get to everything. And that's just completely unacceptable. But I really had to step back and think, if we want to do this work and we want to do it well, you have to make a choice. And so I think in order to make that choice, you have to really get into an in-depth analysis of the student results and each grade and subject each grade band and let that sort of drive where you begin this journey. But then there's a couple of things that I think you have to ask yourself. So really there are probably a lot of options, but two big ones to think about. One, is there a place where you already have some coherence and so you think, okay, if I start there, it's not going to be as heavy of a lift and we can get one subject in grade band up and running really well and let that sort of serve as an exemplar as we go to the more-difficult grade bands and subjects. Or I guess the other way to look at it is: Do I find the place that's most in the ditch and do I tackle that first? So I think, first, it's just understanding that you do have to make a choice whether you want to or not, and then deciding, you know, do we go to the place that really needs it the most, or do we really try to get to a place that's going to be really a quick win for us to get things up and running? Hmm.

Julia Kaufman

Thank you. Yeah, that all makes sense. And I think one of the questions that was asked by one of our participants is just something I just want to chime in on too, about. You know, why does it start to come first? Because that's where you're picking your North Star. And I think that could be a possible strategy before you assemble your team to think about what your North Star is. We're not trying to be too prescriptive here, so you have to think about what makes sense for you and your school. At the same time, there are a lot of pieces in step two that you probably want to have a team weigh in on. So let me talk about that a little bit. So this step is about identifying that North Star. This is where you decide on your focus within that subject and grade levels. You have worksheets and guides to help you throughout the step, and you need to think about what that is that North Star is. It could be your academic standards, but if you know your curriculum is deeply aligned with your standards, potentially the curriculum is your North Star, but there definitely could be others. And then the next piece of that is thinking about what core components of your system send messages about what to teach and how to teach that you need to think about in terms of their alignment with that North Star. So the things we've already talked about standards, curriculum, material, professional learning opportunities, teacher collaboration, time, teacher evaluation assessments. But there may be other things that are really important in your school. Maybe you have a mastery grading system that you want to keep in mind for all this because it's going to affect everything in every single component. Or maybe there are deep community and parent partnerships that you also want to ensure are aligned. So there are other things that could be part of your system of core components, but the ones that we named are ones that are generally in all school systems and are important to consider first. So I'll ask Emily to give her thoughts on what the North Star components would be in their school district and how they differ, maybe across schools in Louisiana from her perspective.

Emily Howell

Okay. For us, our Northstar was curriculum and our main goal, like I mentioned before, was to create masters of that curriculum. So a model that we have in our district is we have content facilitators who are under the chief academic officer and we have school content leads at every school. And the model is that the district academic team works with these school content leads we who then work with teachers. So we train our coaches for that work. The priority is to send those clear messages about what to teach and how to teach it. And just a couple of examples on what that actually looks like. As we support coaches and teachers, we have academic websites for all for content areas. We have grade level leaders that are who are teachers that help us manage those sites. We provide pacing guides, course planners at each content at grade level. We have district grading guidance that we train the coaches and teachers on. We also have protocols such as unit unpacking lessons, study protocol, and student work analysis protocols that are the focal points for teacher collaborations. And we give a lot of guidance about what those teacher collaborations should look like so that we're all saying the same things. Another thing that we do that's a little bit unique, I've not really heard of another district doing it this way. The content facilitators do teach part time, and so each year we kind of set our eyes on a grade that really needs some help and we will grade drop, grade jump to help with those materials and implementation and supporting those teachers directly. And so I think the model of everybody saying the same thing, it's working because people understand what they're supposed to teach and how they're supposed to teach it. In terms of how this could differ across Louisiana, I think that a lot of districts have a lot of priorities, and so sometimes there's confusion on what should be prioritized.

Julia Kaufman

Yeah, that all makes sense to me too. And I feel like a lot of a lot of teachers would would say the same thing as you in terms of like, I need I need some guidance. That's that's going to be helping me tie all these things together. Now this is the data collection piece of this whole tool. It's about you've done the work of kind of thinking about what needs to be coherent with what. And now you're using the tools provided by the toolkit to rate coherence. You're not doing a big survey in this toolkit. What you're doing as a team is thinking about coherence together. So we provide a worksheet for you, and this is what that worksheet looks like. So you can see here, I think you can probably see, I guess it depends on the size of your screen. You can see on the top and the bottom are those core components that you will have identified. So of course they'll be standards, instructional materials, professional development, teacher collaboration, teacher evaluation, summative assessment, interim or benchmark assessments, and then others potentially if you've identified them. And then what we suggest is that each member of the team goes home, takes this worksheet and tries to think about the ratings that they would give for each pair. So the first step in this process is really thinking about, okay, this component instructional materials, what are the key messages that that component conveys about what to teach and how to teach it? You don't need to get into fine grained detail. What we mean by that is just thinking, okay, what are my instruction materials that give me messages? Well, in the case of the teachers we shared, it's EL education, mainly is the thing that gives them messages. If you're thinking about professional opportunities, professional development opportunities, you're thinking about, well, what what really is the focus of most of my professional development? So thinking about that first and then providing a rating for each pair. So for each pair we suggest that you rate coherence using a key that's relatively simple. So a zero to 2 rating, right? Zero would be they don't provide messages about what to teach and how to teach, which is hopefully not the case most of the time. One would be the two components provide somewhat dissimilar or conflicting messages, and a 2 would be they provide similar or reinforcing messages. So give yourself a give your peers a rating and then write down something about why you gave that rating. So in this case, we have an example of someone that gave a 2 for the relationship between standards and materials, and the rationale could be something as simple as, "Curriculum materials mostly support our standards." However, it's unclear whether they support the career-ready standards recommended. So in this example, it's just a short rationale. So you're not making too much work for yourself, but you're just giving a thought, giving a rating and a rationale for each of these pairs. Another thing you can do also is do a similar exercise, but focused on a particular student's subgroup. That's really important in your school system. So let's say you've had an influx of English learners. You've begun to incorporate instructional materials that are specifically for them. You might want to go through the same exercise, except think about English learner or are the English instructional materials for English learners, and the standards are similar to similar? What about professional learning? The professional learning that we get about English learners? Is it addressing the curriculum? Are they are they related to each other or are they not related? So we recommend maybe just thinking about one subgroup, but you could repeat it for special education students or students of a particular subgroup that you really care about your students of color. There could be limitless subgroups. But in the toolkit we say just do one to start and see if that if that really surfaces some really useful things. You know, think about another group that you care about deeply, too. Now I'm going to ask Dana to weigh in on this and thinking about how her district system historically has been more or less coherent to get you all thinking about what in my system is less or more coherent.

Dana Talley

So I want to say first, when I saw this step, I thought, okay, this seems a little tedious, but as I started going through it like this step is very, very necessary. So the first thing I want to say is don't skip this, even though you may feel some need to to maybe just skip right on over and move on to step for this is a very important sort of moment of reflection for you and your team. So when I worked at the State Department, I supported 34 districts and at sort of that 10,000-foot view, it was very obvious to to spot sort of issues with coherence, that there were especially just really obvious issues with coherence. When I returned to my home district in 2020 to serve as chief academic officer, on the surface, things looked really, really good in my parish. And so I, you know, I knew that we had high-quality materials in the hands of teachers. I knew that there was some quality professional, digital-learning opportunities for teachers. And so even though I didn't have this tool at the time, I knew now coming in, in this position, I was going to have to look beyond the surface level and dig in a little bit. So the first thing I want to say is curriculum is our North Star in our parish. There never has been and never could be any other North Star. But curriculum. That curriculum obviously has to be aligned to our state standards, which our Department of Education gives us a rating system so that we can know that or we can do a local view to figure that out and that. And then those, of course, standards are aligned to our end-of-year assessments. So with that out of the way, I wanted to know were the high-quality materials that we had in the hand of hands of teachers actually being used effectively. I wanted to know if the professional learning and teacher collaboration time was really helping teachers become masters of the curriculum. I wanted to know if the formal and informal observations that our leaders were providing for teachers actually included feedback to the teachers on their use of the curriculum. On whether or not the students were doing the work of the curriculum, on whether or not teachers were giving feedback to students on the work of the curriculum. Whether or not we were using curriculum, embedded assessments, and then figuring out what students were, what content students were mastering or not, and then what we were doing about it. And so what I found as I started digging in is we had a little bit of coherence in all of the components. So what I mean by that is we had professional learning opportunities in some grades in subjects that were tied to the curriculum. We had teacher collaboration time in some grades and subjects that were used for unit unpacking and lesson study. We had observations by leaders that were focused on the use of the curriculum some of the time. But what I needed for this district was for that summer to become all, and we wanted consistency across the board with all of those things. And one other thing I think to keep in mind is, you know, it was important as I was trying to figure this out, that I heard from people at all different levels because, you know, people at central office would say to me, "Well, I've said this and I've told them this." But then when I go out and talk to leaders or teachers, that's not what they heard. So I think it's important that you, as you're trying to figure out, like what coherence really looks like in your district. You're not only thinking about the messages you send out, but how people have actually received those messages.

Julia Kaufman

Thank you. That's so powerful. And I feel like what what needs to happen in these groups is for teachers to be present, too, so that teachers can sort of, you know, a principal might say, oh, I think this is this way, but a teacher might give you feedback. This is actually what I'm hearing often times is this. And so having those honest conversations could be a real challenge for everyone, but so important for really understanding what coherence looks like. And in actuality, what Dana talked about, about everything being relatively similar when we collect national data, that's the feedback we get. Yeah, they're pretty similar. We don't get a lot of feedback. The average is not that everything is reinforcing the averages. Everything is similar. And I think as Dana pointed out, you know, getting to that next level to reinforcing is the thing that she pushed. And then I think by creating these tools for you, we're trying to help you reflect on and then potentially move to publishing. So lastly, there's this fourth step, which is creating the improvement plan. So this is the actual work of doing what you think needs to be done in order to improve coherence. So using the data that you've collected from one another, you'll be thinking about what areas really stand out. And that's kind of what Dana did when she got to Lincoln Parish. She collected all this data by talking to people and learning and then figuring out what areas to improve. And then we have some guidance for you to work with your team to brainstorm improvement activities and create a plan. We also have a piece that's about sharing the plan with stakeholders and the district administrators that reviewed our tool kit for us as it was in the development stage, noted that this is a really important piece. Dana was actually one of those reviewers, and this is the piece where you're getting feedback from others on what your plan is and making sure those others know what's happening. So, for instance, if you're focused on middle school math and you have a small team working on this coherence idea, of course you want to share that plan with all the teachers in your school, especially if some math teachers weren't included in the original team. You want to include all teachers, anyone else that could be impacted, maybe even parents, and be really genuine and authentic about gathering that feedback and then really thinking about it and whether it should lead to amendments of your plan. Another thing to think about is, of course you'll want to focus on the areas of conflict you've identified, but also, of course they'll be areas that are within your control, in areas that are outside of your control. If your state assessment is really badly aligned with the teacher evaluation tool that your state also provides. There might not be a lot you can do about that. But so focusing on the things you can do in your system and maybe it is the case that you would say to yourselves, "Well, we'll use the state evaluation tool, but there are other things we can also use alongside of it." So yes, I think that might be in our locus of control in some ways. But think about the things that you can actually control and actually undertake. So those are the big things that we wanted to emphasize to you all for this step, but I wondered if Dana could share a little bit about what her experience with school Improvement has taught about what to keep in mind during this step.

Dana Talley

Yeah, for sure. So I just want to give a little bit of context in our district. So when I worked for the State Department for 15 years, my primary role was to go around and advise superintendents and central staff and principals around school improvement efforts. And then when I had the chance to come back to Lincoln Parish in 2020, you know, that was a little bit of pressure on me. Okay. Dana, can you take all that advice that you've been giving people and can you actually put it into action and will it actually work? And so just to share our data over the last couple of years, we have had growth across the board in proficiency and growth in almost every single subgroup for the last two years in a row. This past year, we had the most growth of any district in the state, and we're number seven in proficiency. So I believe this is not about what I've done. It's about it's about this coherent system that me and everybody else that works in this district have decided to commit to. And so in terms of school improvement and what it's taught me, there are really four things here that I would say. The first is you have to develop your message, your why, what are your core beliefs? And then whatever those core beliefs are, I would say to I would say every single day to every single person I came across. So for us, it was all student should have access to grade level learning every day. All educators and staff can be successful in supporting students with access to grade level learning, and school leaders can and must support educators in meeting the needs of every student. So I said that every day to every person. And now, almost three years later, they're saying that back to me. So really grounding yourself in what do you believe in, in terms of curriculum being our North Star, what makes it possible for all students to have access to grade level learning? Every day is a high quality curriculum aligned to grade level standards. That's why that is our North Star, because it directly ties to our beliefs. So then the second thing I would say is you have to create a sense of urgency. My superintendent gave me grief when I first came. He's like, you know, we're not doing that poorly. Like, why are you harassing folks all the time? Like, let's just, you know, let's just move along. We're doing well. But that's not good enough. Until all students are at grade level, that is not good enough. So you've got to come in even in districts where things are seemingly going well. You have got to create a sense of urgency and then again, help folks understand the reason we're doing these things. The reason we have these school improvement efforts is because we believe all students should have access to grade level learning. All educators can be can support students. All leaders can support educators. And then you've got to get a core group of folks at every level. So central office admin teachers at all levels, and have them really buy in to this idea and help spread the word and help create a sense of urgency. I think Kotter talks about building a coalition. The third thing is, and this is very important. I learned this when I worked at the State Department. You've got to keep it simple. So you're going to develop this plan ever how many pages it turns out to be. But when you are communicating this plan, it absolutely cannot be more than one page. Like in my district, it's a joke. Like we do not put out anything if it's more than one page. Some folks try to put something on front and back and still call it a one pager. But but listen, I'm telling you, all of you folks that are ELA folks and like all these words, those of us that are math folks, we do not like all of those words and we will quit reading. You have got to keep it simple. You've got to keep it to one page. It's got to be something that all stakeholders can easily remember and easily communicate out. And then the last thing I would say, number four, and this is the lesson that I've learned most recently, is you do have to monitor this or monitor this often, but you have got to be ready to adapt when you get credible feedback that something isn't working. So if teachers are coming to you and say, "Hey, there's a problem with this part of the curriculum," or "There's something wrong with this assessment," or whatever, or like "This training is not supporting me well enough to be able to do this." You absolutely have got to listen, get the right people together to figure out if you've got to make some adjustments. And that's something that honestly, when I came into the district, I was hesitant to do and it took me this is now my third year to start doing it. And I wish I had done it from the very beginning because it's made all the difference and teachers feeling like we hear them. Their opinion matters. We value their input and we do. They're the ones working with kids every day. So again, those are the four things that I would say, just I've learned the most about school improvement in my 27 years in this business.

Julia Kaufman

Dana, we've already gotten a couple Q&A questions about like could you could you share those beliefs with with the audience? One person asked, could you share your four core beliefs in the chat? And another said, Dana's messages and lessons are so clear and great. Can we go to one future where they're still learning the message? I would like that too. Can we have that?

Dana Talley

Yes. Is it okay if I if I put it together and send it to you and then you send it to the folks who were on the call today?

Julia Kaufman

If you want to do that, that would be fabulous. Thank you. There is another question in the in the in the chat and the Q&A that I'd like to raise. Are there district policy shifts that have occurred to support the sustainability of working in a more coherent way, for example, curriculum, adoption policies, stipulations for collaborative time and structures in your bargaining agreements? If not, do you think that is a necessary step in the process? Yeah. Dana, you're nodding your head.

Dana Talley

So so we don't we don't really have teacher unions here. So I know that that's different than a lot of places. We didn't really we didn't put this in policy. Our our board in general, you know, we're careful about putting everything into policy. You may have a district where it needs to be in policy, but we just set expectations with principals. You will include common planning time in your master schedule. You will, during that time be focused on, you know, unit unpacking, lesson study analysis of work within the curriculum. So these are really just in our district. We set it up as expectations. And I also super, I'm a small district. We have 12 schools, so I realize that's not helpful to some of you out there. But I also supervise principals, which is helpful because when when we set an expectation at central office, you know, I'm in schools all the time. I have monthly check-ins with principals and we can make sure that that our expectations are actually being carried out. So we didn't really change policy, but we did make our expectations very clear.

Julia Kaufman

Mm hmm. Yeah. Thanks. And the other thing that I think to highlight is that there are states working to ensure.

Dana Talley

That.

Julia Kaufman

Districts are able to identify and select high-quality materials standards, online materials. In our study of Louisiana, they were among the first state to really focus on this and focus on districts. Here are all the things that we have reviewed really carefully and teachers have helped us review and that we think are really aligned with our standards. And there's a whole group of people in the council chiefs, state school officers called the High Quality Instructional Materials Professional Development Network. There's a group of state officials that are really focused on ensuring that high-quality materials standards, aligned materials are being provided as options and making creating all kinds of incentives and ways so that those can be adopted more easily. So that's one example of a kind of policy that makes coherence a more sustainable proposition.

Dana Talley

And Juliet, can I can I add to that? I really think and I was at the State Department when we first started implementing this, but that's really key. Like, if districts don't have that, it's really hard for them to say, okay, yes, we're going to now choose one of these curricula that's been rated high quality. I really feel like districts need the state to step up and say, like, we expect you to have aligned materials. That's the only way we can ensure students have access to grade level learning every day.

Julia Kaufman

Hmm. Yeah. And, Emily, I don't want to put you on the spot here, but I'm wondering, as a teacher, when you're thinking about these high-quality materials and their availability to you, and I know a lot of teachers really like to also create some of their own materials, especially in English language arts. I'm wondering for you, how do you see the provision of curriculum versus being able to create your own materials for students?

Emily Howell

I think just over the years I've realized that teachers need a curriculum that's been coherently created for a purpose aligned to standards. I think teachers appreciate more when they when they can zoom out and look at the K-12 staircase and look at how that all works together. And so one thing I really try to do as much as I can as a facilitator is really let grade levels work with each other, not just similar grade level, but fifth with fourth, fifth with ninth. We trace the standards across to grade levels so that people can see it's less about doing a fun activity that's enjoyable in your classroom and more about giving the students a coherent curriculum.

Julia Kaufman

Mm hmm. Yeah. Thanks. Yeah, I think that comes up a lot for us when we're collecting our data and thinking about high-quality materials. I appreciate you giving your perspective as well. I'm going to ask, we don't have any more questions, the Q&A. So I'm going to ask just one more question to the panelists. And if anything else comes up, we will raise that as well. But for the panelists, as you're thinking across all these steps, I'm just wondering if anything pops up as well. This is going to be particularly challenging for most school systems. If there's any step or particular task that you feel like could be particularly hard. Something to put a lot of thought around and reflection around before putting it out there for a team to undertake.

Emily Howell

I think I could answer part of this question, Dana spoke about. We're three years in at this point on our improvement plan, but having that urgency and just not giving up on your messaging, like sometimes you feel like you've said something over and over and over and it's it's at that point when it's really starting to land. And it was at that school professional development when I heard a teacher say everyone is saying the same thing, that I realized the message is starting to land. So just to to be diligent and to persevere in continuing that coherent message.

Dana Talley

Yeah. I would piggyback on what what Emily said. I do think it's, you know, I will tell you, and I'm. I tend you probably figured this out, but I tend to be a little bit of a of a bulldozer coming in. And I, I try to build good relationships. But again, I want to create that sense of urgency because someone's got to create the sense of urgency. There's got to be one person in your district who has the ability to hold people accountable that will create this sense of urgency. So I do I do think that's really important. I, I think also what's going to be really hard is deciding where to start. I think you're going to have a lot of people with a lot of ideas. And I think you have to decide for yourself, like, where does it make sense to start? Like for one, I don't think you want to start in a place that's going to be so hard and potentially is going to take so long, and then people aren't going to necessarily even have a positive feeling about it when it's over. Like you really have to think about where's the right place to start? And just the same as you said, like don't give up. Like I, I was met with such resistance when I came just because people didn't, you know, we're doing fine. Like, what's the problem? Why are you complaining that our data is not good? But we have all persevered and now there is little to no resistance with the work that we're doing. It's just what we do in Lincoln Parish. So I would just encourage you, you know, take the time, decide what your focus is going to be on, and then just persevere exactly as Emily said.

Julia Kaufman

I love that a lot of your recommendations focus on communicating. And making sure you're communicating, for example, urgency, but also thinking a lot about what you're communicating and how it's perceived. That feels like a really important part of the process from leadership, for sure. Another question that was asked in the chat is, is the superintendent the only or the right person to create that sense of urgency?

Dana Talley

So I'm not the superintendent. I'm the chief academic officer. But I will tell you, I'm smart enough to not go out and say anything that I haven't vetted with my superintendent. So then when people hear it from me and they go around to him, he's like, "Oh, yeah, Dana and I have already talked about that and we're on the same page." So I, you know, for me, I supervise principals. It's my job to make sure that pre K-12 in terms of curriculum instruction is is aligned. But I do think if it if it started at the top of the superintendent like there's nothing wrong with that. But I don't think it has to be the superintendent. I do think that person has to to be willing to back you when it when it's not easy.

Julia Kaufman

Yeah. So making sure that the superintendent is aware of what you're doing and backs you up when needed. But it's not necessarily that one person that has to do everything. Yeah, that makes a lot of sense. There is another question. Are there any metrics that you think are important to monitor to know if this is working? For example, I wonder if increased teacher use of materials is an outcome of improved coherence. I also wonder if there are student experience or outcome metrics that would relate to any of these improvements. Oh, that's such an important question, I feel like, and something we didn't address really deeply in the toolkit. But something as you're improve your identifying your improvement activity is it's so important to try to figure out how do you know if it's working right. It would be interesting to hear from Dana or Emily on this. If you did identify an improvement activity that was really about. Ensuring the professional development was better aligned with curriculum. How would you know if the things you were doing were working? I mean, I think in that case I always think about data collection, right? Like talking to teachers after a professional development and trying to understand the extent to which they think that that professional development was aligned with curriculum or also making sure to check if every professional development offered to teachers in your system does integrate curriculum in some way. I don't know if there are other things that occur to you guys, Dana and Emily.

Dana Talley

Yeah, I'm going to let Emily start. Emily, I'll add to you have something to say or I'm welcome. I'm happy to start if you want me to.

Emily Howell

And I think that that's one of the mistakes we've made over the past couple years in terms of professional development. I kind of stepped back and had to ask myself, like, does this fit with our strategic plan, our priorities? And some of the sessions didn't. And so this past year, we made a more concerted effort like, if it's not part of our priorities in our strategic plan, like we're not doing it and we have to make sure that those things match. And so, I mean, that didn't really have a metric. I just sort of realized that was incoherent. So I don't know, Dana, if you have anything to add there.

Dana Talley

Yeah, I think what I would add is like, what matters to me is what I observe in the classroom and then, you know, student results on especially curriculum embedded assessments. So we use a walkthrough tool that actually we created at the State Department a number of years ago. But it looks for three things. It looks for the teachers use of the curriculum. It looks at whether or not kids are engaged in the work of the curriculum. So the thinking, the talking, the work, and then it looks at if teachers are giving feedback to students on the work of the curriculum. And so as we see issues, as we're observing teachers using that tool, then that helps decide like, "Okay, well, what maybe needs to happen at our next unit unpacking or what needs to happen the next time we get all of these grade levels together," because we're seeing either maybe a misunderstanding of the curriculum or just they need more practice with this component of lessons and that sort of thing. So those are the things we really try to look at. But I will say we do give surveys and we do listen like like we hear from people whether we want to or not. And if if we're hearing a lot of rumbling that folks are unhappy, like we pay attention to that and we try to get to the source of that and try to see, is it just a misconception on their part about the curriculum or is it really something that we need to go in and make some adjustments around?

Julia Kaufman

Thanks to all that. And there is one other question just briefly before I wrap this up. What our success indicators. Could you give some examples of some of the kinds of success indicators that you might have been referencing? And I think this is the question potentially for Emily, but I'm not sure I.

Emily Howell

Can say two things there. I mean, we always look at our assessment data. It speaks volumes. Assessment index, proficiency index and looking at cohorts across the years, I think that tells us a lot about success. But just kind of anecdotally, we hear from students and they say things like, "We've been doing this since the third grade," and I'll say, "Have you seen this? The reader circles?" "We started seeing those in elementary school." And so I hear things from students all the time that that really helps me to know that our system's coherent. But again, that's a little bit more anecdotal.

Julia Kaufman

Yeah, I think that's really helpful. And I think it probably depends a lot on the things that you have defined as success. What what does success look like? And then trying to think about what would tell you what success looks like. I'm just going to finish this up with a few notes about our project and the research we're doing. You can find out more about that research at this URL here. This is a page that includes all of our instructional research, including our toolkit. And you can see here we've done a quite a lot over the years, and we have a forthcoming publication that focuses specifically on Tennessee and Rhode Island as to case studies for thinking about coherence. But we've done a ton of other work to really research this construct of coherence and why it matters. So we encourage you to take a look. I think that we've reached the end of the webinar. I just want to express my deep gratitude to Dana and Emily for being with us today and for sharing their experiences. It's been really enlightening. We really appreciate your time. Again, we will have this recorded. We are having it recorded and we'll provide that recording on our website when it's ready to go. But feel free to email us with any questions. Our emails are easy to find on the RAND website as well as in the invitation that you RSVP to. So thank you so much for your time.

Research conducted by

This report is part of the RAND presentation series. RAND presentations may include briefings related to a body of RAND research, videos of congressional testimonies, and a multimedia presentation on a topic or RAND capability. All RAND presentations represent RAND's commitment to quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.