The Effect of Panel Membership and Feedback on Ratings in a Two-Round Delphi Study

Past observational studies of the RAND/UCLA Appropriateness Method have shown that the composition of panels affects the ratings that are obtained. Panels of mixed physicians make different judgments from panels of single specialty physicians, and physicians who use a procedure are more likely to rate it more highly than those who do not. This paper sets out to determine the effect of using physicians and health care managers within a panel designed to assess quality indicators for primary care and to test the effect of different types of feedback within the panel process. This paper provides further experimental evidence that consensus panel judgments are influenced both by panel composition and by the type of feedback which is given to participants during the panel process. Careful attention must be given to the methods used to conduct consensus panel studies, and methods need to be described in detail when such studies are reported.