Reliability of Ratings of the Scoop Notebook and Transcripts

The previous paper in this symposium provided background information on the Scoop research project and the approach we are using to try to develop rich descriptions of classroom practice in middle school mathematics and science (Creighton, 2005). The researchers collected data from 36 middle school science classrooms, including “Scoop” notebooks of artifacts collected by teachers, transcripts of audiotapes of classroom discourse, and direct observations of instruction. A framework for judging eleven dimensions of “reform oriented” classroom practice was developed, along with a detailed scoring guide to use to rate each type of artifact. These dimensions were used to rate observations, notebooks and transcripts. In addition, readers of notebooks and transcripts rated each source of information on “completeness” on the same one-to-five scale, and assigned a “confidence” score to their overall set of ratings. This presentation will focus on the procedures that were used to rate the notebooks, transcripts, and classroom observations and the reliability of the judgments that were made.