Methodological Challenges Associated with Patient Responses to Follow-Up Longitudinal Surveys Regarding Quality of Care

Published in: Health Services Research, v. 38, no. 6, pt. 1, Dec. 2003, p. 1579-1598

Posted on on January 01, 2003

by Katherine L. Kahn, Honghu H. Liu, John L. Adams, Wen-Pin Chen, Diana M. Tisnado, David Carlisle, Ron D. Hays, Carol Mangione, Cheryl L. Damberg

Read More

Access further information on this document at

This article was published outside of RAND. The full text of the article can be found at the link above.

OBJECTIVE: To illustrate, using empirical data, methodological challenges associated with patient responses to longitudinal surveys regarding the quality of process of care and health status, including overall response rate, differential response rate, and stability of responses with time. DATA SOURCES/STUDY SETTING: Primary patient self-report data were collected from 30,308 patients in 1996 and 13,438 patients in 1998 as part of a two-year longitudinal study of quality of care and health status of patients receiving care delivered by 63 physician organizations (physician groups) across three West Coast states. STUDY DESIGN: The authors analyzed longitudinal, observational data collected by Pacific Business Group on Health (PBGH) from patients aged 18-70 using a four-page survey in 1996 and a similar survey in 1998 to assess health status, satisfaction, use of services, and self-reported process of care. A subset of patients with self-reported chronic disease in the 1996 study received an enriched survey in 1998 to more fully detail processes of care for patients with chronic disease. DATA COLLECTION/EXTRACTION METHODS: The authors measured response rate overall and separately for patients with chronic disease. Logistic regression was used to assess the impact of 1996 predictors on response to the follow-up 1998 survey. They compared process of care scores without and with nonresponse weights. Additionally, the authors measured stability of patient responses over time using percent agreement and kappa statistics, and examined rates of gender inconsistencies reported across the 1996 and 1998 surveys. PRINCIPAL FINDINGS: In 1998, response rates were 54 percent overall and 63 percent for patients with chronic disease. Patient demographics, health status, use of services, and satisfaction with care in 1996 were all significant predictors of response in 1998, highlighting the importance of analytic strategies (i.e., application of nonresponse weights) to minimize bias in estimates of care and outcomes associated with longitudinal quality of care and health outcome analyses. Process of care scores weighted for nonresponse differed from unweighted scores (p<.001). Stability of responses across time was moderate, but varied by survey item from fair to excellent. CONCLUSIONS: Longitudinal analyses involving the collection of data from the same patients at two points in time provide opportunities for analysis of relationships between process and outcomes of care that cannot occur with cross-sectional data. The authors present empirical results documenting the scope of the problems and discuss options for responding to these challenges. With increasing emphasis in the United States on quality reporting and use of financial incentives for quality in the health care market, it is important to identify and address methodological challenges that potentially threaten the validity of quality-of-care assessments.

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.