Read Online

Validation Study of the TNTP Core Teaching Rubric

by Andrew McEachin, Jonathan Schweig, Rachel Perera, Isaac M. Opper

Download eBook for Free

FormatFile SizeNotes
PDF file 0.5 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. To what extent does content expertise of the rater influence scores on TNTP Core?
  2. To what extent are teacher observation scores (overall and on each of the four rubric domains) valid predictors of the effectiveness of math and ELA teachers, as captured by their estimated contributions to student performance (or value-added scores)?

Nearly every public school teacher in the United States receives some form of feedback based on classroom observations, and these assessments are used by policymakers and administrators as indicators of teacher quality and input for professional development purposes. The TNTP Core Teaching Rubric, introduced in 2014, is unlike others in that it is scored based on student behavior rather than teacher actions. In this report, RAND researchers assess whether the TNTP Core Teaching Rubric produces scores that are representative of teachers' overall instructional practices and whether raters' content expertise influence scores on TNTP Core.

For this study, volunteer teachers of mathematics and English language arts (ELA) from more than 20 districts or charter school networks across eight states had their classroom instruction videotaped three times. This sample was complemented with a sample of approximately 100 4th-and 5th-grade teachers drawn from the Measures of Effective Teaching study for whom videotaped instruction was available. Raters trained in the TNTP Core Teaching Rubric then scored the videos from both samples. Analyzing the raters' scores, RAND researchers found a modest relationship between teachers' TNTP Core scores and student achievement gains in ELA, and no statistically or practically significant relationships between TNTP Core scores and math achievement gains. Further, raters struggled to agree on their judgments about the quality of a lesson, suggesting a considerable amount of uncertainty about the extent to which TNTP Core scores represent teachers' overall instructional practices. The report includes a number of recommendations to improve the use of TNTP Core.

Key Findings

  • Relationships between teachers' TNTP Core scores and student achievement gains are modest and vary by subject area.
  • Observers often disagree in their ratings of instructional practice based on the TNTP Core rubric, and these disagreements may be related to their content expertise.
  • There is a considerable amount of uncertainty about the extent to which TNTP Core scores represent teachers' overall instructional practices.

Recommendations

  • Consider setting higher standards for rater certification (specifically by requiring raters in training to have their scores align with those of master raters at a higher frequency).
  • Research has shown that raters tend to change their approach to scoring over time — a phenomenon known as rater drift. To mitigate this, consider using more frequent post-certification calibration and validation exercises during a rating period.
  • Keep in mind that high-quality evaluation and feedback require many observers with different backgrounds to rate many lessons.
  • Consider collecting additional sources of evidence that support claims about the quality of teacher practice, i.e., using TNTP Core as one measure in a multiple-measure system.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Design of Study

  • Chapter Three

    Methods

  • Chapter Four

    Consistency, Bias, and Reliability

  • Chapter Five

    Extrapolation and Predictive Validity

  • Chapter Six

    Conclusion and Recommendations

  • Chapter Seven

    Limitations

Research conducted by

The research described in this report was prepared for TNTP and conducted by RAND Education.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.