This evaluation describes the Regional Choice Initiative (RCI), a large-scale initiative in 17 school districts in Pennsylvania. The RCI sought to expand school choice and offer more-rigorous courses for students in grades 7 to 12. RAND evaluated the performance of the individual programs and the RCI overall. The authors also identified aspects of implementation that facilitated or hindered the performance of the RCI.
Evaluation of the Regional Choice Initiative
- To what extent did each RCI program achieve its performance and objective measures?
- What factors enabled or hindered each RCI program's achievement of performance measures?
- To what extent did the overall RCI affect student engagement and achievement outcomes?
School districts in Beaver County, Pennsylvania, have been working to further improve student achievement by increasing students' exposure to more-rigorous courses. In 2007, the Beaver Valley Intermediate Unit (BVIU), a regional educational-service agency, received funding for a five-year grant under the U.S. Department of Education's Voluntary Public School Choice program. The BVIU responded by developing, implementing, and evaluating the Regional Choice Initiative (RCI), a large-scale initiative designed and implemented in 17 school districts. The RCI sought to expand school choice — as well as provide opportunities for students in low-performing districts to learn in high-performing environments — for students in grades 7 to 12 by offering four programs: Open Seats, Dual Enrollment, Cyber Learning, and Academies for Success. The RCI was implemented for six years — 2007–2008 to 2012–2013.
The BVIU commissioned RAND to conduct a formative and a summative evaluation of the RCI programs. The RAND team took a mixed-methods, quasi-experimental approach to evaluate the implementation and impact of the RCI initiative. The authors reviewed program documents, interviewed a number of stakeholders throughout the region, surveyed parents, and analyzed student RCI participation data and administrative data. The RAND team also worked with the RCI team and partnering superintendents to develop the RCI logic model and set up performance measures for each RCI program to guide the implementation and evaluation of the RCI. RAND evaluated the performance of the individual programs and the RCI overall. The authors also identified aspects of implementation that facilitated or hindered the performance of the RCI.
The Overall Participation in RCI Programs Was Low and Inconsistent
- In total, 4,357 students participated in four RCI programs. The number of participants increased from 165 in year 1 to 2,063 in the final two years.
- The number of participants each year was less than 10 percent of the overall enrolled students in grades 7–12 in the first four years of program implementation.
- Most participants (86 percent) participated in RCI for only one year, and more than 90 percent of participants took courses through only one RCI program.
Although RCI Met Most of Its Objective Measures, It Failed to Meet the Majority of Its Performance Measures
- The RCI achieved eight out of 21 measures.
- Five of the eight achieved measures were objective measures. The three achieved performance measures were about the dropout rate of participating students and the number of courses in the core curriculum created by content-area faculty teams.
- None of the performance measures that required a certain percentage of participating students earning a passing grade in courses enrolled through RCI programs was met.
Data Available for Impact Analysis Were Limited; No Relationship Was Detected Between Participation and Student Engagement and Achievement Outcomes
- Participating districts were unable to provide all the data required for analysis. Insufficient data limited the statistical power of the analyses and therefore the ability to identify program effects.
- The analysis based on available data did not find any discernible relationship between participation in RCI programs and student achievement and engagement outcomes at the school or district level.
Multiple Factors Contributed to the Low Level of Participation in the RCI
- These factors include lack of quantifiable indicators for eligible students; school stakeholders' concerns about the potential impact of the program on students, staff, and the school; lack of compatible schedules across schools; students' unwillingness to attend courses using virtual-learning technologies; and logistical constraints for students to physically transfer among schools to take courses.
Stakeholders Viewed the Impact of the RCI on Participating Districts and Students Positively, and Districts Decided to Sustain Most RCI Programs in the Future
- Stakeholders reported that the RCI and its programs provided important opportunities to students.
- Stakeholders reported that the RCI and the work of its administrators improved communication and collaboration between districts and transformed the climate and culture of the county.
- Test the feasibility of designed program features and the implementation plan in advance. This feasibility study should examine the demand for the expected program, examine the readiness and capacity of participating organizations to implement the program features, and identify problems that might arise during the implementation of the program.
- Create a strong, multitiered communication strategy to communicate program information to the target student population and their parents. Experts in communication might support the development of effective campaigns that appeal to the target population and encourage recruitment.
- Use a dashboard system from the beginning of the program to monitor the implementation and facilitate communication with stakeholders.
Table of Contents
Regional Choice Initiative
Methods, Data, and Analysis
Overview of RCI Participation
Performance of Open Seats
Performance of Dual Enrollment
Performance of Cyber Learning
Performance of Academies for Success
Overall Performance of the RCI
Relationship of Participation in the RCI to Student Outcomes
Key Findings and Recommendations
Student Outcome Analysis Methods and Results