Transforming an Urban Public School District

Illustration by Vilisov/iStock

New Haven Public Schools (NHPS) is an urban school district in Connecticut serving approximately 21,000 students in 46 schools, with nine high schools. Concerned that only about one-half of its students were meeting state proficiency standards in reading and math tests or graduating within four years of starting high school, NHPS and the City of New Haven set out to design and implement new measures to improve the school system with the hope of producing skilled, critical thinkers ready for careers and strengthening New Haven's economic development and sense of community.

The School Change Initiative (School Change), announced in 2009 by the City of New Haven and NHPS, set out to eliminate the achievement gap, cut the high school dropout rate, and ensure that students are prepared for and financially able to attend and succeed in college. School Change's efforts are being implemented incrementally over five years, with the goal of having a fully established education reform in the 2015–2016 school year.

The New Haven Promise (Promise) scholarship program was announced in 2010 by the City of New Haven, NHPS, and the Community Foundation for Greater New Haven, with financial support from Yale University, to increase the enrollment and success in college of NHPS high school students. It seeks to cultivate a college-going culture, build community and parental engagement, and promote New Haven's economic development by offering capable local students financial and social support to attend college. Promise complements the district's School Change reform by providing incentives for students to dedicate themselves to their studies (via academic eligibility requirements) and to contribute to their community (via community service requirements). The high school graduating class of 2011 was the first cohort eligible for the scholarship.

In 2013, the Board of Directors of New Haven Promise asked the RAND Corporation to assess the short-term progress made by School Change and Promise. RAND's study sets the foundation for evaluations that could occur in later years to determine the reforms' effectiveness. This approach allows time for organizational conditions to be put in place, instruments to be validated, data systems and internal dashboards to be established, and the reforms to take hold. The RAND study also supported internal monitoring efforts that Promise and the district have instituted; in the coming years, Promise will report key educational outcomes annually to the public.

Although it is too early in the programs' implementation to fully assess their impact, the preliminary analysis suggests that students are making progress on key indicators, while critical challenges remain.

Some outcomes showed promise

Reports of school climate were positive. Starting in spring 2010, NHPS administered the School Learning Environment (SLE) survey to assess students', teachers', parents', and school staffs' perspectives of school climate by asking them for their perceptions of their school's orderliness, safety, discipline, facilities, and nonacademic measures of social relationships and school connectedness. Studies have shown that schools that have a healthy school climate also have students who are engaged in their studies and have positive levels of achievement. RAND researchers examined changes in perspectives on school climate from student and teacher SLE survey responses and found that, on average, students and teachers rated their schools favorably, with students' and teachers' responses improving significantly between the 2009–2010 and 2010–2011 school years. However, there was little measurable change in subsequent years. (Although the SLE survey is also administered to parents, response rates were too low to provide a valid assessment of their perceptions.)

Student achievement scores gained slightly, especially for the schools deemed most in need. Following the implementation of School Change, overall NHPS scores in both math and reading assessments of the Connecticut Mastery Test (CMT) and the Connecticut Academic Performance Test (CAPT) increased somewhat, making them comparable to schools and districts across Connecticut with similar sociodemographic and academic profiles but still far below the Connecticut state average. Although schools designated as the most in need (tier III) continued to be the lowest performing, these schools yielded the largest gains in both reading and math test scores, relative to forecasts RAND conducted based on prereform trends. This suggests that schools that were targeted for improvement by School Change's school turnaround efforts were making steady and significant progress.

Dropout rates improved. On average, the dropout rate of students who started high school in the 9th grade and dropped out in the 10th grade improved substantially, from 9 percent for the 2008–2009 cohort to just 3 percent for the 2011–2012 cohort. Schools whose rates improved were more likely to be tier III and have larger percentages of disadvantaged students than other schools. However, it is not clear what impact, if any, the School Change or Promise programs had on dropout rates because rates also improved steadily at schools in other Connecticut public school districts with similar sociodemographic and academic profiles.

College attendance rates rose steadily. For all NHPS high school graduates, enrollment in college after graduation has been steadily increasing. For the graduating class of 2013, those who met the three Promise scholarship eligibility requirements RAND examined (a grade point average [GPA] of 3.0 in high school, continuous enrollment, and 90 percent attendance), college attendance increased by 6 percentage points over the graduating class of 2008 (although this number still represented only 36 percent of graduates).

Photo by monkeybusinessimages/iStock

Opportunities to improve remain

Changes in personnel that occurred in the first years of School Change prompted mixed opinions among students and parents. While Promise Scholars and parents of Promise Scholars noted that personnel changes had affected school climate in both positive and negative ways, particularly in the restructuring of school administration in the lowest tier schools, many felt that changes in staffing caused confusion and did not necessarily improve instruction, classroom practices, or a school's climate.

District and school engagement with parents was deemed lacking. Parents of Promise Scholars felt that schools and districts did not engage with them directly or want their input. As an example, few parents in the study's focus groups completed the SLE surveys after the first year, citing concerns about whether the district read their responses, given that most reform elements had already been planned or designed by the time the first SLE survey was administered (in 2009–2010).

Student achievement scores remained, on average, low and fell below the state average. Although there was some progress in CMT math scores among elementary and middle school students, gains were small. Furthermore, reading scores did not improve, and overall, NHPS students still lagged considerably behind the rest of the state in both math and reading.

Preparation for college provided to students could improve. All Promise Scholars in the study noted barriers to their college-going efforts, such as a lack of information from the district and Promise regarding college or the Free Application for Federal Student Aid application process; confusion about Promise eligibility requirements and funding distribution processes; and a lack of streamlined or accurate reporting on required data for the Promise application, particularly attendance, community service hours, and GPA. Parents felt high school counselors could provide better advice and be more available to answer questions. Promise Scholars also indicated that they struggled with study skills, time management, and self-discipline, which many felt they had not learned in high school. Some noted that, even with the advent of School Change, they did not feel their high school experience adequately prepared them for college-level coursework.

Graduates of NHPS high schools struggled to meet Promise eligibility requirements. Some troubling figures about NHPS high school graduates remain: Fewer than one-half achieved a 3.0 GPA or higher, and those who met the three Promise requirements that RAND analyzed were less likely to be minorities, less likely to be economically disadvantaged, and less likely to be classified as English-language learners or have special needs.

Looking ahead

The study summarized in this brief is a first step in a broader evaluation effort. The baseline trends this study analyzed provide evidence to help School Change and Promise officials and stakeholders make midcourse corrections to improve the programs' success and to leverage areas that are working well. Tracking baseline trends sets the foundation for evaluations that can occur in later years, thereby allowing the NHPS Board of Education, the Promise Board of Directors, and the broader New Haven City community to determine whether the reforms are succeeding. Given the many components of School Change and Promise that are intended to work together to bring about districtwide transformation, gathering and reporting on key outcomes and broad community-level impacts can support analyses of how well NHPS is faring as the programs evolve and grow over time. To complement the RAND study's findings, the district and Promise can undertake internal efforts and evaluations to guide continuous improvement.

Internal reporting mechanisms improve decisionmaking. In the coming years, Promise plans to report progress of the district and Promise in a web-based tool or internal "data dashboard." This web-based report to the public will track the status of key activities and outcomes each year. It will present summary statistics graphically, using easily understood charts. Annual progress reports will support transparency and engagement between the district and Promise and the broader stakeholder community. This type of annual reporting will allow funders of the Promise and School Change initiatives, parents, teaching and administrative staff, and district central office and Promise staff to ascertain what kinds of incremental changes are being made over time. Tracking the progress of key indicators through time also helps ensure that viable data systems and linkages among data sets are in place for future evaluations.

Evaluations inform improvements and determine reforms' effectiveness. While it is too early to tell whether School Change and Promise are meeting their goals, the study reported in this brief found indicators that show some early progress is being made. Because educational reform outcomes often take several years after full implementation to demonstrate results, further research will be needed to determine the programs' long-term success.