Cover: Validity Evidence Supporting Use of Anchoring Vignettes to Measure Teaching Practice

Validity Evidence Supporting Use of Anchoring Vignettes to Measure Teaching Practice

Published in: Educational Assessment, Volume 24, Issue 3, pages 155–188 (2019). doi: 10.1080/10627197.2019.1615374

Posted on Feb 8, 2022

by Julia H. Kaufman, John Engberg, Laura S. Hamilton, Kun Yuan, Heather C. Hill

High-quality measures of instructional practice are essential for research and evaluation of innovative instructional policies and programs. However, existing measures have generally proven inadequate because of cost and validity issues. This paper addresses two potential drawbacks of survey self-report measures: variation in teachers' interpretation of response scales and their interpretation of survey questions. To address these drawbacks, researchers tested out use of "anchoring vignettes" in teacher surveys to capture information about teaching practice, and they gathered validity evidence in regard to their use as a tool for adjusting teachers' survey self-reports about their instructional practices for research purposes, or potentially to inform professional development. Data from 65 teachers in grades 4–9 responding to our survey suggested that vignette adjustments were reliable and valid for some instructional practices more than others. For some instructional practices, researchers found significant and high correlations between teachers' adjusted survey self-rating, through use of anchoring vignettes, and previous observation ratings of teachers' instruction, including ratings from several widely-used observation rubrics. These results suggest that anchoring vignettes may provide an efficient, cost-effective method for gathering data on teachers' instruction.

Research conducted by

This report is part of the RAND external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.