Reliability of Ratings of the Scoop Notebook and Transcripts

by Brian M. Stecher

Full Document

FormatFile SizeNotes
PDF file 0.2 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

The previous paper in this symposium provided background information on the Scoop research project and the approach we are using to try to develop rich descriptions of classroom practice in middle school mathematics and science (Creighton, 2005). The researchers collected data from 36 middle school science classrooms, including “Scoop” notebooks of artifacts collected by teachers, transcripts of audiotapes of classroom discourse, and direct observations of instruction. A framework for judging eleven dimensions of “reform oriented” classroom practice was developed, along with a detailed scoring guide to use to rate each type of artifact. These dimensions were used to rate observations, notebooks and transcripts. In addition, readers of notebooks and transcripts rated each source of information on “completeness” on the same one-to-five scale, and assigned a “confidence” score to their overall set of ratings. This presentation will focus on the procedures that were used to rate the notebooks, transcripts, and classroom observations and the reliability of the judgments that were made.

The work reported herein was supported under the Educational Research and Development Centers Program and conducted within RAND Education.

This report is part of the RAND Corporation working paper series. RAND working papers are intended to share researchers' latest findings and to solicit informal peer review. They have been approved for circulation by RAND but may not have been formally edited or peer reviewed.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.