The Intensive Partnerships for Effective Teaching initiative, which was funded by the Bill & Melinda Gates Foundation, was a multiyear effort to improve student outcomes by increasing student access to effective teaching. In this report, the authors discuss the challenges in defining metrics and collecting data and describe how the RAND team addressed those challenges.
Using Data to Support the Intensive Partnerships for Effective Teaching Initiative
Data Collection, Metric and Dashboard Creation, and Lessons Learned
Download eBook for Free
| Format | File Size | Notes |
|---|---|---|
| PDF file | 0.9 MB | Use Adobe Acrobat Reader version 10 or higher for the best experience. |
Research Questions
- What key data-related challenges can be identified, and what recommendations can address those challenges?
- What lessons can be learned related to the systematic use of education data for periodic program monitoring?
The Intensive Partnerships for Effective Teaching initiative, which was funded by the Bill & Melinda Gates Foundation, was a multiyear effort to improve student outcomes—particularly high school graduation and college attendance among low-income minority students—by increasing student access to effective teaching. The RAND Corporation worked with the foundation to collect and warehouse data from participating sites and to produce annual data dashboards that presented quantitative information about key indicators of the progress of the reforms. During the course of this project, the RAND data team conducted four key activities: (1) defining the metrics that would be used to monitor and assess annual progress and that would appear in the dashboard, (2) collecting the data from the sites to compute the metrics, (3) managing and standardizing the data, and (4) creating the dashboard and reporting the metrics to the sites and the foundation.
This report discusses the challenges in defining metrics and collecting data. It also describes how the RAND data team addressed those challenges. Specifically, the authors examine challenges and recommendations in four areas: (1) issues related to defining metrics used to track system performance; (2) issues related to data collection; (3) issues related to managing and standardizing data across sites; and (4) issues related to data confidentiality, data sensitivity, and partnerships. The authors also draw overarching lessons related to the systematic use of education data for periodic program monitoring.
Key Findings
Long-term projects need the flexibility to accommodate change
- During this project, there were changes in sites' data collection procedures, data management systems, and research emphasis.
Success might depend on establishing good relationships with counterparts in district offices
- Concerns about accurate data handling are essential to a successful data-based analytic project, but it might be equally important to invest in building positive working relationships with client data administrators and information technology personnel. Many times, progress occurred because of the strong professional connections developed with counterparts in the sites.
Redefining and rescaling existing measures required creative thinking to define comparable metrics across sites
- Scaling and categorizing questions surfaced repeatedly in different contexts. Different kinds of solutions worked in different contexts, and a successful data-based analytic project will benefit from creative thinking about ways to rescale or reclassify based on existing measures.
Collaboration with district staff was critical in the deidentification of student data and the substitution of new unique identifiers
- Confidentiality requirements necessitate data use agreements and data safeguarding procedures that will add time and complexity to the project.
Recommendations
- To help solve challenges related to developing metrics, work with local administrators to understand conditions in the schools to ensure that measures will be compatible.
- To help solve challenges related to developing metrics that examine multiple entities, balance the metrics between those that allow for comparisons across entities and those that are more valid for examining trends within a single entity.
- To help solve challenges related to data collection, communicate with site administrators often to become aware of potential changes that might influence analysis or reporting and inform stakeholders about how such changes will affect reporting.
- To help solve challenges related to the standardization and management of data, encourage the adoption of standards and gather information and context on how and why data were collected; it might be necessary to adjust analyses and reports to correspond to available data.
- At the outset of a project, determine the types of data that are available and try to develop metrics and analytic methods that reflect the available options, but recognize that there are no simple ways to address some of the challenges related to collecting these types of data.
- To help solve challenges related to data safeguarding and privacy, make personal visits to each site to establish working relationships with data providers, develop mutually agreeable procedures, and build confidence in researchers' ability to handle sensitive data.
Research conducted by
The research described in this report was conducted by RAND Education and Labor and supported by the Bill & Melinda Gates Foundation.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
This work is licensed under a Creative Commons Attribution 4.0 International License. All users of the publication are permitted to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.