Download
Download Free Electronic Document
| Format | File Size | Notes |
|---|---|---|
| PDF file | 2.5 MB | Use Adobe Acrobat Reader version 10 or higher for the best experience. |
Purchase
Purchase Print Copy
| Format | List Price | Price | |
|---|---|---|---|
| Add to Cart | Paperback16 pages | $20.00 | $16.00 20% Web Discount |
This Note describes methods for measuring rater agreement and making inferences about the accuracy of dichotomous ratings from agreement data. The first section presents a probability model related to latent class analysis that is applicable when ratings are based on a discrete trait. The second section extends these methods to situations in which ratings are based on a continuous trait, using a model related to signal detection theory and item response theory. The values obtained by these methods provide either direct or upper-bounds estimates of rating accuracy, depending upon the nature of the rating process. Formulas are shown for combining the opinions of multiple raters to classify cases with greater accuracy than simple majority or unanimous opinion decision rules allow. Additional technical refinements of the probability modeling approach are possible, and it promises to lead to many improvements in the ways that ratings by multiple raters are analyzed and used.
This report is part of the RAND Corporation Note series. The note was a product of the RAND Corporation from 1979 to 1993 that reported other outputs of sponsored research for general distribution.
This research in the public interest was supported by RAND, using discretionary funds made possible by the generosity of RAND's donors, the fees earned on client-funded research, and independent research and development (IR&D) funds provided by the Department of Defense.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.