The Effect of Panel Membership and Feedback on Ratings in a Two-Round Delphi Study

Results of a Randomized Controlled Trial

Published in: Medical Care, v. 37, no. 9, Sep. 1999, p. 964-968

Posted on on January 01, 1999

by Stephen M. Campbell, Mark Hann, Martin Roland, Julie Ann Quayle, Paul G. Shekelle

Read More

Access further information on this document at

This article was published outside of RAND. The full text of the article can be found at the link above.

Past observational studies of the RAND/UCLA Appropriateness Method have shown that the composition of panels affects the ratings that are obtained. Panels of mixed physicians make different judgments from panels of single specialty physicians, and physicians who use a procedure are more likely to rate it more highly than those who do not. This paper sets out to determine the effect of using physicians and health care managers within a panel designed to assess quality indicators for primary care and to test the effect of different types of feedback within the panel process. This paper provides further experimental evidence that consensus panel judgments are influenced both by panel composition and by the type of feedback which is given to participants during the panel process. Careful attention must be given to the methods used to conduct consensus panel studies, and methods need to be described in detail when such studies are reported.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.