Hours of Opportunity

Out-of-school-time (OST) programs, which include after-school and summer learning programs, are increasingly recognized as factors that can help children and youth develop into healthy, productive members of society. However, OST programming within cities can be fragmented and uncoordinated, resulting in poor quality and insufficient access for those most in need of services. To address these issues, The Wallace Foundation established an initiative in 2003 to help five cities develop better coordinating mechanisms to reduce OST fragmentation, redundancy, and inefficiency and to increase OST access and quality. The initiative provided grants to Providence, Rhode Island; New York City; Boston; Chicago; and Washington, D.C. A key requirement of the initiative was for the cities to build an information, technology, and communication infrastructure to facilitate the management and support of OST programs. In 2008, The Foundation asked RAND to assess the progress of these five sites, including their development and use of management information (MI) systems to track participation in OST programs. The second in the resulting three-volume series examines the adoption of MI systems in the five grantee cities, plus three others—Denver, Louisville, and San Francisco—which were added to represent sites that had not been part of the initiative.^[1]

Lessons for an Emerging Field

While each of the sites developed and implemented its MI system in a different way, many of the factors that constrained or enabled the systems' use were common across sites. The shared experiences point to several lessons for other cities interested in improving OST provision through the use of data.

MI systems give cities much-needed data about OST programming and participation, which can be used to improve programs. All the cities used data from their MI systems to better understand OST programming and participation (e.g., enrollment, attendance, and demographics and program characteristics). The importance of this use cannot be underestimated. Prior to MI system adoption, the cities could say little about the programs they funded or the youth being served. City officials put this information to work to improve participation and programming.

MI systems must be carefully planned to support system improvement. While planning to implement MI systems, cities in the study faced many decisions that affected the ways in which the systems could be used later. The lesson is that a clear understanding of the goals for the data, including how those goals support larger OST system building, is a prerequisite for an effective MI system. In addition, it might be necessary to modify the MI system based on experience to enhance its utility. Likewise, cities had to adopt methods of breaking down barriers to using the system—through training, allocating staff time to data analysis, or hiring external evaluators.

MI systems can improve contract management and enable a greater focus on the quality of programming. Prior to implementing an MI system, two cities described the management of OST providers as strictly contractual and focused on "paper" rather than programs. MI system data enabled them to focus on program quality, allowing city managers to use average daily attendance reports to identify potential quality problems and then communicate with providers to determine whether they needed assistance.

Using data to showcase OST efforts can lead to additional funding and support. City leaders and OST providers reported that the ability to show their funders enrollment, participation, and outcome data led to increases in resources and provided them with a competitive advantage over others seeking funding.

Customized web-based systems encourage MI system use. Most MI system users interviewed found the systems easy to use and said that the web-based interface, implemented in all the cities in the study, encouraged timely data entry because the system could be accessed from multiple locations. Moreover, customization of the MI systems to meet the specific needs of city agencies and providers led to greater use of the systems and their data.

Investing in high-quality training reaps benefits. All cities offered training to providers and city-level program officers to familiarize them with the MI system. However, providers who reported receiving high-quality training were more likely than others to believe that the city MI system was useful. Providers' demand for training was high, and they particularly wanted more advanced training in data analysis and the use of the data.

Mechanisms to eliminate redundancies in data entry and reporting requirements would help providers. One of the most significant constraints on providers' use of MI system data was the burden of entering the same data into multiple MI systems, which could include the city's MI system, another city's MI system, the provider's MI system, and a private funder's MI system. One solution would be better coordination across major city organizations and implementation of a common MI system. Another option would be to configure the city's MI system to allow providers to enter additional data required by other funders or their own organizations. A third option is to allow providers to easily exchange data between the city's OST MI system and other MI systems that providers use.

Mayoral demand for data encourages cities' use of the data. Without a demand for data at the highest levels, some city officials found it difficult to give priority to analyzing data even when they desired to do so. However, mayoral demand for data encouraged city officials to set aside time and prioritize activities and resources in order to analyze, report, and use MI system data.

MI Systems Can Help Improve OST Provision

Developing MI systems and using the resulting data for decisionmaking are new practices in the OST field. Although many of the cities in the study did not consider their MI system work to be complete, they all acknowledged that simply having data on the programs and the youth served is an important first step. City officials and OST providers reported a shared vision that the information collected by MI systems will support OST system improvement. Overall, the adoption of MI systems appears to be a positive step for the field and will likely improve OST programs.

Notes

• [1] The first volume in the series, Hours of Opportunity, Volume 1: Lessons from Five Cities on Building Systems to Improve After-School, Summer School, and Other Out-of-School-Time Programs, describes the sites' work under the grant and analyzes the conditions and activities that contributed to the cities' progress.
• The third volume, Hours of Opportunity, Volume 3: Profiles of Five Cities Improving After-School Programs Through a Systems Approach, presents case studies of the five Wallace initiative cities.

Topics

• After-School Programs
• Boston
• Chicago
• District of Columbia
• Education Administration
• Extracurricular Activities
• New York City
• Nonformal Education
• Providence