Direct Measurement of Three Stylistic Dimensions of National Institutes of Health Consensus Statements

The Consensus Development Program of the National Institutes of Health (NIH) publicly evaluates scientific information concerning biomedical technologies. Since 1977, the program has issued over 80 consensus statements on a wide range of medical topics. An earlier RAND content analysis of 24 consensus statements showed that consensus statements varied in three ways: (1) discursive statements attempt to present a full description of the technology and the context surrounding its use; (2) directive statements attempt to formulate the findings of the conference in terms of guidelines for health care practitioners; and (3) scholarly statements attempt to review the research on the technology. This Note considers whether the qualities of discursiveness, directiveness, and scholarliness can be directly measured by a straightforward rating scale. Results showed that straightforward ratings only moderately replicated the content analysis; however, these ratings had an internal reliability and consistency. Therefore, NIH should not abandon direct ratings. If direct ratings are to be obtained, they should be done by group discussion rather than by individual raters. Raters should be at least partially familiar with the entire set of medical technologies to be rated and should be comfortable with the concept of Likert-type rating scales.