The Intensive Partnerships for Effective Teaching initiative, which was funded by the Bill & Melinda Gates Foundation, was a multiyear effort to improve student outcomes by increasing student access to effective teaching. In this report, the authors discuss the challenges in defining metrics and collecting data and describe how the RAND team addressed those challenges.

Using Data to Support the Intensive Partnerships for Effective Teaching Initiative

Data Collection, Metric and Dashboard Creation, and Lessons Learned

Gerald P. Hunter, Stephanie Williamson, Asa Wilks, Janet M. Hanley, Brian M. Stecher

ResearchPublished Mar 5, 2020

The Intensive Partnerships for Effective Teaching initiative, which was funded by the Bill & Melinda Gates Foundation, was a multiyear effort to improve student outcomes—particularly high school graduation and college attendance among low-income minority students—by increasing student access to effective teaching. The RAND Corporation worked with the foundation to collect and warehouse data from participating sites and to produce annual data dashboards that presented quantitative information about key indicators of the progress of the reforms. During the course of this project, the RAND data team conducted four key activities: (1) defining the metrics that would be used to monitor and assess annual progress and that would appear in the dashboard, (2) collecting the data from the sites to compute the metrics, (3) managing and standardizing the data, and (4) creating the dashboard and reporting the metrics to the sites and the foundation.

This report discusses the challenges in defining metrics and collecting data. It also describes how the RAND data team addressed those challenges. Specifically, the authors examine challenges and recommendations in four areas: (1) issues related to defining metrics used to track system performance; (2) issues related to data collection; (3) issues related to managing and standardizing data across sites; and (4) issues related to data confidentiality, data sensitivity, and partnerships. The authors also draw overarching lessons related to the systematic use of education data for periodic program monitoring.

Key Findings

Long-term projects need the flexibility to accommodate change

  • During this project, there were changes in sites' data collection procedures, data management systems, and research emphasis.

Success might depend on establishing good relationships with counterparts in district offices

  • Concerns about accurate data handling are essential to a successful data-based analytic project, but it might be equally important to invest in building positive working relationships with client data administrators and information technology personnel. Many times, progress occurred because of the strong professional connections developed with counterparts in the sites.

Redefining and rescaling existing measures required creative thinking to define comparable metrics across sites

  • Scaling and categorizing questions surfaced repeatedly in different contexts. Different kinds of solutions worked in different contexts, and a successful data-based analytic project will benefit from creative thinking about ways to rescale or reclassify based on existing measures.

Collaboration with district staff was critical in the deidentification of student data and the substitution of new unique identifiers

  • Confidentiality requirements necessitate data use agreements and data safeguarding procedures that will add time and complexity to the project.

Recommendations

  • To help solve challenges related to developing metrics, work with local administrators to understand conditions in the schools to ensure that measures will be compatible.
  • To help solve challenges related to developing metrics that examine multiple entities, balance the metrics between those that allow for comparisons across entities and those that are more valid for examining trends within a single entity.
  • To help solve challenges related to data collection, communicate with site administrators often to become aware of potential changes that might influence analysis or reporting and inform stakeholders about how such changes will affect reporting.
  • To help solve challenges related to the standardization and management of data, encourage the adoption of standards and gather information and context on how and why data were collected; it might be necessary to adjust analyses and reports to correspond to available data.
  • At the outset of a project, determine the types of data that are available and try to develop metrics and analytic methods that reflect the available options, but recognize that there are no simple ways to address some of the challenges related to collecting these types of data.
  • To help solve challenges related to data safeguarding and privacy, make personal visits to each site to establish working relationships with data providers, develop mutually agreeable procedures, and build confidence in researchers' ability to handle sensitive data.

Topics

Document Details

Citation

RAND Style Manual
Hunter, Gerald P., Stephanie Williamson, Asa Wilks, Janet M. Hanley, and Brian M. Stecher, Using Data to Support the Intensive Partnerships for Effective Teaching Initiative: Data Collection, Metric and Dashboard Creation, and Lessons Learned, RAND Corporation, RR-2917-BMGF, 2020. As of October 11, 2024: https://www.rand.org/pubs/research_reports/RR2917.html
Chicago Manual of Style
Hunter, Gerald P., Stephanie Williamson, Asa Wilks, Janet M. Hanley, and Brian M. Stecher, Using Data to Support the Intensive Partnerships for Effective Teaching Initiative: Data Collection, Metric and Dashboard Creation, and Lessons Learned. Santa Monica, CA: RAND Corporation, 2020. https://www.rand.org/pubs/research_reports/RR2917.html.
BibTeX RIS

Research conducted by

The research described in this report was conducted by RAND Education and Labor and supported by the Bill & Melinda Gates Foundation.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This work is licensed under a Creative Commons Attribution 4.0 International License. All users of the publication are permitted to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.