A New Instrument to Measure Appropriateness of Services in Primary Care

OBJECTIVE: To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits. DESIGN: Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits. SETTING: Primary care practices in a staff model health maintenance organization and a large preferred provider network. MEASURES: Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic. RESULTS: For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3% of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56% for new medication, 31% for diagnostic test, and 22% for referral. CONCLUSIONS: This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.