Designing and Sustaining Innovative High Schools

Photo by Monkey Business Images/Adobe Stock

Focusing on mastery, personalized learning, and social and emotional learning is increasingly common in high schools across the United States. What if these practices were the primary focus of a school's design? How would they be implemented? Would student outcomes improve?

The recent experiences of 16 small high schools of choice in large urban districts across the nation offer some important insights about the successes, challenges, and outcomes associated with enacting complex, comprehensive school design changes. The Carnegie Corporation of New York's (CCNY's) Opportunity by Design (ObD) initiative, launched in 2013, was based on the premise that promising high school reform strategies need to be fully integrated into school design and supported by appropriate and sustained levels of financial, policy, and implementation supports. The goal of the initiative was to create high schools designed to help underprepared students catch up and graduate in four years with the academic and social and emotional skills needed for postsecondary success.

This research brief summarizes the final report of RAND Corporation researchers who studied the initiative from 2014 through 2018. The report focused on estimating the effects of the ObD initiative on student academic achievement, attendance, and suspension outcomes and describing implementation in the final year of the initiative. The report also compared ObD teachers' reports of their instructional practices with those of high school teachers nationally. The findings and recommendations from this study may be of interest to schools and districts seeking to design, implement, and sustain innovative high school models to promote equitable learning opportunities and increase the achievement and graduation rates of traditionally underserved students.

About Opportunity by Design

Sixteen high schools in six large, urban districts and one charter management organization (CMO) participated in the ObD initiative. The six districts were Cleveland, Ohio; Denver, Colo.; New York, N.Y.; Philadelphia, Pa.; Prince George's County, Md.; and Providence, R.I. The CMO was located in Brooklyn, N.Y. Table 1 further describes the participating schools.

The ObD schools received funding and other supports to adopt ten design principles, developed by CCNY, in ways that best fit their unique environments and continuously refined them over the course of the initiative (see text box). CCNY funded and launched Springpoint: Partners in School Design, a national school design organization, to support ObD school and district leaders in designing, establishing, and sustaining innovative high schools.

The schools' instructional models were grounded in three design principles that reflect key aspects of innovative instructional approaches; these are shown in bold in the text box. CCNY highlighted these three design principles as foundational to the schools' models, and they are thus emphasized in RAND's report.

How Did the ObD Schools Implement Mastery, Personalization, and Positive Youth Development Practices?

To answer this, the team surveyed ObD teachers, students, and administrators. The team also visited each school to interview teachers and administrators and conduct focus groups with students.

The intent of mastery-based learning differed from how it was implemented in practice. Mastery-based learning generally focuses on supporting deep understanding and application of knowledge by allowing students to move through content at their own pace and attempt a task or assignment multiple times. Staff at half of the ObD schools described mastery as students' deep understanding of a topic, while others defined it as the completion of tasks or assignments. Regardless of how schools conceptualized mastery, the most common mastery-based practice was providing students with multiple attempts to demonstrate mastery.

Personalization focused on a relatively small number of practices. Personalizing learning often involves tailoring learning experiences to students' individual learning needs and interests and offering students extensive choices of content, topics, or learning materials. In all ObD schools, personalizing learning entailed tailoring instructional practices, materials, and topics to accommodate students' interests, but offering extensive choice was uncommon in most schools.

Most schools emphasized positive youth development (PYD) skills to support student academic achievement and positive behavior, but these skills were often not explicitly taught or assessed. PYD is closely related to social and emotional learning, which fosters interpersonal (e.g., teamwork) and intrapersonal (e.g., self-discipline) capabilities that help students succeed in school and life. Most ObD schools had structures and systems to help students build social and emotional skills that could help improve their academic achievement and build a positive school culture but did not always make explicit connections between these skills and academic content.

Support for innovation and autonomy from some district policies and structures facilitated implementation. ObD principals who described receiving more district-level support were members of strong district-wide networks of innovative schools. All ObD schools had the autonomy to adopt or develop curricula and select or design professional development (PD) aligned to their school models. Staff at most schools reportedly used a blend of instructional materials, combining those they developed or sourced with others provided by external partners or the district. In most ObD schools, principals developed their own PD materials or worked with external providers.

Springpoint, a technical assistance provider, played a unique and important supporting role. CCNY designed the initiative to provide schools with multiple years of Springpoint support, which generally focused on building the capacity of the founding principals, teachers, and select district staff. Nearly all school leaders appreciated the coaching and feedback Springpoint provided during school visits and the opportunity to connect with leaders at other ObD schools, as well as with external support providers.

How Were ObD Teachers' Instructional Practices Different from Those of Other U.S. High School Teachers?

Mastery, personalization, and PYD practices are increasingly common in schools across the United States. To better understand how ObD teachers' reports of their instructional practices compared with those of high school teachers nationally, the research team conducted a nationally representative survey using RAND's American Teacher Panel (ATP).

ObD teachers reported more extensive use of practices related to mastery, personalization, and PYD than high school teachers nationally. Figure 1 shows ObD teachers' responses to three survey items that exemplify each of the three key design principles and compares them with the responses of high school teachers nationally. Responses to the full set of survey items can be found in the main report.

Access to frequent, varied data was likely key to implementing mastery-based learning practices. When compared with teachers across the nation, ObD teachers reported more-frequent receipt of data on the performance of individual students, including data about students who needed extra assistance and students who had achieved mastery. Data focused on individual students can help teachers tailor their instruction to individual students' learning needs, so it is noteworthy that ObD teachers reported greater access to this information.

Survey questions	ObD	ATP
Mastery: Students can make multiple attempts at a given task that counts toward mastery.**	92	64
Personalization: I provide a variety of materials or instructional approaches to accommodate individual needs and interests.**	86	76
PYD (SEL): I connect academic content to themes related to social and emotional learning.**	80	70

NOTES: **indicates statistically significant difference in item means at alpha < 0.01. ATP results are weighted to be nationally representative. This figure presents the two most positive responses on a four-point response scale. Responses shown for the personalization and PYD items are "I emphasize this practice to a moderate extent," and "I emphasize this practice to a large extent." Responses shown for the mastery question are "This practice resembles my practices to a moderate extent" and "This practice resembles my practices to a large extent." Mastery N: ObD = 164, ATP = 811; Personalization N: ObD = 180, ATP = 998; PYD N: ObD = 180, ATP = 998. SEL = social and emotional learning.

Did the ObD Initiative Improve Student Outcomes?

The study investigated whether the ObD initiative affected student academic, behavioral, and college readiness outcomes in the fourth year of the initiative. Table 2 summarizes the outcomes examined.

Analysis of each outcome compared the ObD students with a comparison group of non-ObD student peers. The comparison groups were constructed to be as similar to the groups of ObD students as possible, using available information. For most measures, comparison students attended non-ObD high schools in the same districts; for the MAP measures, comparison students were drawn from a national database.

Figure 2 shows the results of the outcomes analyses for the whole ObD initiative in the Overall column, and then for each of the districts individually. The figure shows standardized effect sizes both numerically and with bar graphs, with zero (no effect) indicated by the black vertical line. Statistical significance is indicated by pink shading for negative effects and green shading for positive effects; otherwise, the bars are gray.

Overall, the evaluation found no positive effects on student outcomes. The Overall column in Figure 2 shows that there were no significant positive effects of the initiative, with two significant negative effects. The negative credit accumulation effect was driven by results in District 2. SAT estimates were more uniformly negative across districts.

For individual districts, estimated effects on academic outcomes tended to be positive as measured by MAP but negative on state assessments. The state assessments measured attainment of grade-level content standards, while the MAP measured student learning growth even if not aligned to grade-level standards. Consequently, these results could reflect implementation of personalized and mastery-based learning, where ObD schools may have placed relatively greater emphasis on student growth, even if off-grade-level, and less emphasis on attainment of grade-level standards.

All data from 2017–2018 except where noted

Academic outcome	Overall	District 1	District 2	District 3	District 4	District 5	District 6	District 7
State mathematics assessment (algebra) or PSAT8/9, grade 9	–.17	–.17, statistically significant	.02	–.28, statistically significant	–.15, statistically significant	–.52, statistically significant	.00	–.02
State ELA assessment or PSAT8/9, earliest grade tested 9–11	–.05	–.12, statistically significant	–.07	.02	.00	–.05	.08	–.16, statistically significant
Mathematics growth (MAP grades 9–11)	.05	–.18	.01	.11, statistically significant	insufficient data	.03	.11, statistically significant	.03
ELA growth (MAP grades 9–11)	.04	–.84, statistically significant	–.06	.04, statistically significant	insufficient data	.11	.01	.16, statistically significant

Behavioral outcome	Overall	District 1	District 2	District 3	District 4	District 5	District 6	District 7
Attendance	–.03	.46, statistically significant	.07	–.15, statistically significant	–.21, statistically significant	–.07, statistically significant	insufficient data	.00
Suspensions (reduction displayed as positive)	.03	.26, statistically significant	–.11, statistically significant	.17, statistically significant	–.08, statistically significant	.04, statistically significant	insufficient data	–.13, statistically significant

College readiness	Overall	District 1	District 2	District 3	District 4	District 5	District 6	District 7
PSAT mathematics + verbal, grade 10	–.03	–.02	–.05	insufficient data	–.10, statistically significant	.04	insufficient data	–.03
SAT mathematics + verbal, grade 11 or 12	–.08, statistically significant	–.09, statistically significant	–.01	insufficient data	–.17, statistically significant	–.04, statistically significant	insufficient data	–.05
Credit accumulation	–.27, statistically significant	.10	–1.65, statistically significant	.06, statistically significant	insufficient data	.00	insufficient data	–.12
AP tests passed	.00	insufficient data	insufficient data	insufficient data	.24	–.19, statistically significant	insufficient data	insufficient data
Graduation 2017–2018	.02	.01	insufficient data	insufficient data	insufficient data	.18, statistically significant	insufficient data	–.15, statistically significant
Graduation 2018–2019	.01	.23, statistically significant	.05	insufficient data	–.43, statistically significant	.20, statistically significant	insufficient data	–.09

NOTE: The horizontal length of the bar represents the magnitude of the standardized program effect estimate, with the vertical lines representing zero. For some extreme estimates, bars are truncated. Bars are green or red where results are statistically significant after correction for multiple hypothesis tests; otherwise they are gray. Results are not shown where insufficient data were available.

What Accounts for These Findings?

Like most studies of broad high school initiatives, the ability of the analysis to capture the full breadth of potential ObD effects was limited. The study measured a broad set of outcomes, but the academic measures available across districts were limited to mathematics and ELA assessments that do not reflect the breadth of subject matter addressed in high schools. To understand the extent to which ObD or other initiatives for high school students are preparing students for postsecondary education or careers, researchers would need to follow students into their postsecondary lives to document post–high school performance.

Such challenges as leadership turnover and inflexible district policies hindered implementation. CCNY designed the initiative to provide schools with multiple years of Springpoint support, which focused on building capacity of the founding principal and select district staff. In some cases, Springpoint did not have the resources to provide the same level of support to new principals or district staff when turnover occurred. As a result, new staff had a limited understanding of ObD design principles and school models. In addition, ObD school and district grading policies were often misaligned, which is a common challenge faced by schools embracing mastery-based instructional approaches.

Conclusion and Recommendations

This study found that the design principles focused on instructional practices were challenging to implement. Thus, staff and students in schools using these practices may need different resources and supports than other high schools. District and school leaders considering similar reforms at scale may wish to consider the following recommendations to enhance implementation.

Make sure the right data and resources for mastery-based instruction are available. The findings suggest that teachers in mastery-based schools need frequent, high-quality, accessible data about student mastery. Principals and district staff should work with teachers to ensure that they can access the data they need easily and frequently and also ensure that assignments and assessments provide detailed information about student mastery. Teachers in mastery-based schools might also benefit from resources, such as easily adapted curriculum materials and assistance from expert organizations, to assist students still behind in mastery.

Partner with teachers to select and/or develop high-quality curricula. Many teachers expressed the desire for expert assistance in developing at least a portion of their curriculum materials related to mastery-based and personalized learning environments. Districts and school principals should consider working together, with teams of teachers and with external support providers (such as Springpoint), to provide resources for curriculum development.

Focus on PYD and related SEL competencies. The extent to which schools focus on such skills as communication, critical thinking, and collaboration can have implications for how well-prepared students will be to achieve their academic and postsecondary goals. Mastery-based schools can consider including competencies for SEL skills in mastery frameworks, providing teachers with research-based curricula and assessments aligned to those competencies, and targeting training to help teachers implement them. Future schools could also consider ways to help students adjust to the expectations of a mastery-based environment.

Be ready for leadership turnover and school change before they occur. Districts, funders, and external support providers could help ease district leadership and principal transitions by planning for these changes early. For example, district leaders and funders could work with principals to develop school operations manuals and provide support targeted to new principals.

Have a dedicated, aligned system of support. Implementing innovative reforms is almost always a complex endeavor. Support from an external organization, such as Springpoint, was perceived as helpful and is not a feature common to most reform efforts. Moreover, principal and teacher need for support may change over time as the program develops. An aligned system of continuous support that includes district leaders, school leaders, and teachers could benefit schools engaged in complex reforms.