MACRA Palliative Care Quality Measure Development—Testing Summary Report">

MACRA Palliative Care Quality Measure Development—Testing Summary Report

Measure Name: Feeling Heard and Understood

by Sangeeta C. Ahluwalia, Brian G. Vegetabile, Maria Orlando Edelen, Claude Messan Setodji, Anthony Rodriguez, Adam Scherling, Jessica Phillips, Carrie M. Farmer, Jordan M. Harrison, Julia Bandini, Wenjing Huang, Lucy B. Schulson, Anne Walling, Sarah Dalton, Monique Martineau, Owen Hall, Danielle Schlang, Melissa A. Bradley, Katherine Ast

This Article

RAND Health Quarterly, 2022; 9(3):3


Palliative care has expanded rapidly in the past 20 years, especially in the ambulatory (office) setting, and there is growing consensus regarding the need to systematically measure and incentivize high-quality care. The Centers for Medicare & Medicaid Services entered a cooperative agreement with the American Academy of Hospice and Palliative Medicine (AAHPM) as part of the Medicare Access and CHIP Reauthorization Act of 2015 to develop two patient-reported measures of ambulatory palliative care experience: Feeling Heard and Understood and Receiving Desired Help for Pain. Under contract to AAHPM, RAND Health Care researchers developed and tested both measures over a three-year project period. Researcher efforts included identifying, developing, testing, and validating appropriate patient-reported data elements for each measure; developing and fielding a survey instrument to collect necessary data in a national beta field test with 44 ambulatory palliative care programs; and collecting and analyzing data about measure reliability and validity to establish measure performance and final specifications. Further, the authors elicited provider and program perspectives on the use and value of the performance measures and their implementation and elicited the perspectives of patients from racial and ethnic minorities to understand their experience of ambulatory palliative care and optimal approaches to measurement. In this study, the authors present results from their test of the Feeling Heard and Understood performance measure, which they demonstrate to be a reliable and valid measure that is ready for use in quality improvement and quality payment programs.

For more information, see RAND RR-A400-1 at

Full Text


Palliative care has grown tremendously in the past 20 years. Once thought to be limited to delivery in hospital or hospice settings, palliative care services can be delivered to patients in ambulatory (office) clinics or as a supplement to ongoing treatment, such as for cancer. At its core, palliative care is patient-centered care, and effective palliative care focuses on patient values and care preferences, symptom management, and peace at the end of life.

Palliative care stakeholders, including patients and their caregivers, providers, and health care systems, have come to see the value in systematically measuring the quality of palliative care. Yet there are few measures endorsed by the National Quality Forum to assess provider performance, and the performance measures that are in use are limited to narrowly defined populations, such as hospice patients. Perhaps surprisingly, given the focus of palliative care on the patient, there are no patient-reported measures of palliative care quality to provide insight into patients’ experiences of care.

The Centers for Medicare & Medicaid Services (CMS) set out to address gaps in palliative care quality measurement, particularly for accountability programs (such as the Quality Payment Program), through a cooperative agreement (1V1CMS331639-01-00) with the American Academy of Hospice and Palliative Medicine (AAHPM) as part of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA). The purpose of this agreement was to

  • develop and test two patient-reported performance measures for patients receiving ambulatory palliative care
  • ensure that the proposed measures incorporate patient voice and preferences and are broadly applicable to patients with serious illness and their families receiving palliative care services in a range of ambulatory primary and specialty care settings
  • convene a technical expert panel that incorporates patient, caregiver, and family input directly into the performance measure development, specification, testing, and implementation processes
  • submit the proposed palliative care measures to CMS's 2021 Measures Under Consideration list for the Quality Payment Program and to the National Quality Forum for review and endorsement so that providers can measure and improve the quality of care in ambulatory palliative care.

AAHPM partnered with RAND Health Care and the National Coalition for Hospice and Palliative Care to develop two patient-reported performance measures of palliative care experience in the domains of symptoms and communication. The symptom measure is intended to assess the extent to which patients who used ambulatory palliative care received the help they wanted for their pain, and the communication measure is intended to measure the extent to which they felt heard and understood by their palliative care provider and team. In this study, we describe the final measure specifications, methods, testing results, and considerations for implementation for the proposed Feeling Heard and Understood performance measure. A companion study covers the same information for the proposed Receiving Desired Help for Pain measure (Ahluwalia et al., 2021).

Methods and Results

The proposed Feeling Heard and Understood performance measure was tested rigorously in a multiphase process. We began by creating a patient experience survey instrument, then tested the wording and structure of data elements with patients who received palliative care in an ambulatory setting. We conducted an alpha test first to test and finalize the survey-based data elements and pilot the survey within five palliative care programs. Subsequently, we tested the performance measure at the national level in a beta field test, which occurred over ten rounds through 44 programs. The national beta test was paused from March to September 2020 because of interruptions in program operations during the coronavirus disease 2019 (COVID-19) public health emergency.

National Beta Test Patient Population and Sampling

Feedback from our technical expert clinical user and patient panel (TECUPP) and project advisory group reinforced the importance of including in the performance measure's target population all patients ages 18 and older who had at least one ambulatory palliative care visit. Working with the Center to Advance Palliative Care and the National Coalition for Hospice and Palliative Care, we identified 395 ambulatory palliative care programs in the United States from which to develop a sampling frame.

We contacted 238 palliative care programs, 70 of which were deemed ineligible to participate in the test for such reasons as not providing ambulatory care or not having practitioners eligible for the Merit-based Incentive Payment System (MIPS). Of the remaining programs, 44 participated by sending at least one sample file during field testing, 39 participated before and after the pause in data collection, three did not continue participation in September 2020, and two programs joined at that time. Ten were hospice sites, 24 were hospitals, and ten were ambulatory or other administrative sites.

Among these programs, we aimed to field 6,000 to 7,500 surveys and receive 2,400 to 3,000 completed surveys from patients receiving ambulatory palliative care. Patient sampling was conducted each month from November 2019 to February 2021, with a pause from March 2020 to September 2020 because of the COVID-19 pandemic. Fielding was conducted in ten rounds of 56 days each, beginning with patient prenotification and ending after 56 days. Patients were first contacted by mail or email to notify them (or their caregivers, if they were too ill to participate) that the survey would arrive soon, then they were mailed or emailed the survey. Patients were given three weeks to respond to the survey and up to eight attempts to respond to the survey through a computer-assisted telephone interview. Patients who did not respond to the survey within 56 days were considered nonrespondents.

Over the course of the national beta field test, the survey instrument changed slightly. The instrument contained 43 data elements at the start of the test. In March 2020, during the sixth round of fielding, five data elements were added regarding the COVID-19 pandemic. When we resumed fielding in September 2020, we revised the instrument to include a total of eight data elements regarding the COVID-19 pandemic.

We fielded the survey to 7,595 sampled patients across ten rounds of data collection. Of the surveys fielded, 3,356 were not returned, 1,435 were excluded from any analyses because of ineligibility for the larger study, and 2,804 were returned and included in analyses. This resulted in a 37 percent raw response rate and a 46 percent response rate when excluding ineligible patients. Responses were distributed evenly across programs. With respect to mode, the majority of returned surveys were completed by mail (as opposed to by phone or web). About 19 percent of surveys were excluded because of ineligibility, such as the patient being deceased or not being involved in completing the survey. Most respondents were female (56 percent), White (88 percent), non-Hispanic/Latino (95 percent), and highly educated. The mean age of respondents was 63 years. One-third of programs each contributed 31 to 50 completed surveys.

Critical Data Element Testing

The construct of feeling heard and understood previously was measured using a single question: “Over the past two days, how much have you felt heard and understood by the doctors, nurses, and hospital staff?” That question was developed for use in the inpatient palliative care setting (Gramling et al., 2016); to improve the reliability of the data element for ambulatory palliative care patients, we evaluated ten data elements for possible inclusion as a scale during national beta testing. These potential data elements addressed feelings of comfort, trust, and value alignment, as well as whether patients feel considered as a whole person in their care. Our TECUPP and project advisory group provided input on the final wording, response options, and inclusion of each data element in the survey instrument. We also evaluated the psychometric properties and the reliability (both internal consistency and test-retest) and convergent validity of the potential data elements, as follows:

  • Survey participants tended to respond at the high end of the response scale.
  • The mean data element–level missingness for the entire survey was low (0.8 percent).
  • There were no discernable patterns among the four data elements that we ultimately selected for the scale. Correlation of data elements ranged from 0.54 to 0.69.
  • Reliability of the four–data element scale was high, indicating that scores obtained are reliable and can, therefore, be used in the construction of the performance measure.
  • High scores on Feeling Heard and Understood were associated with high Consumer Assessment of Healthcare Providers and Systems (CAHPS) communication scores, supporting convergent validity of the data elements.

Performance Measure Testing

We evaluated the Feeling Heard and Understood performance measure first by the proportion of item-level top-box scores (i.e., items for which the most favorable response option was selected) across respondents and programs; then by a set of potential risk adjusters using program information, program data, and U.S. Census data; then by three potential denominator exclusions, two approaches to scoring, statistically significant differences among programs, reliability at the program level, and validity at the program level. We found the following:

  • The proportion of top-box scores across respondents had a mean of 0.71, showing that responses skewed toward the high end of the scale across the four data elements. Variability across programs, however, was high.
  • With one exception (survey mode), all the potential risk adjustment variables were found to have significant or nearly significant associations with programs, which means they could confound the relationship between quality of care and measure score.
  • Analysis of variance by respondent type (i.e., patient-only, proxy-assisted, and proxy-only) found no significant difference between proxy-only and patient-only groups, but there was a significant difference between these two and the proxy-assisted type. In keeping with the spirit of the performance measure, we decided to exclude proxy-only responses but retained proxy-assisted responses.
  • Using survey mode and indicator of proxy assistance as risk adjusters, we found that measure performance adjusted slightly downward for phone-completed surveys and slightly upward for web-completed surveys and for proxy assistance.
  • The distribution of scores showed differences from the mean across programs.
  • Bayesian generalized mixed-effects models showed a reasonable level of reliability based on between-program variability and within-program variability, given our sample sizes. Recalculations according to program size showed some sensitivity to small programs.
  • The Feeling Heard and Understood performance measure was significantly and positively associated with the CAHPS communication measure, supporting the convergent validity of the Feeling Heard and Understood performance measure.

Feasibility Assessment

To assess the feasibility of implementing Feeling Heard and Understood in practice, we interviewed stakeholders to understand (1) costs and burden to programs, (2) programs’ perspectives on implementation, and (3) potential variability in patients’ experiences of care by racial and ethnic minority groups. Our findings are as follows:

  • Using a survey vendor to administer the survey would minimize burden to programs. Most programs have prior experience using such vendors.
  • Quotes from five CMS-approved survey vendors to administer the mixed-mode patient surveys varied from approximately $2,500 to $12,500 per year.
  • Program interviewees reported generally positive perceptions of the Feeling Heard and Understood performance measure, with some concerns, including survey fatigue and understandability of the survey for those with low literacy levels.
  • For quality assessment and improvement efforts, program interviewees discussed examining external factors, such as scheduling and appointment delays, to further assessment.
  • Most programs had little to no experience with providing services over telehealth before the pandemic. Benefits of telehealth included enhanced personal connections (including with family members) and medication reconciliation; challenges included loss of human touch with patients.
  • Patients and family members who identify as Black or African American or as Asian who were interviewed for their perceptions of palliative care indicated (1) a general lack of awareness of palliative care, (2) generally feeling heard and understood by their palliative care providers, and (3) higher satisfaction with palliative care providers than providers in the general health care system.

Implementation Considerations and Conclusions

As part of our examination of the testing and analytic procedures used during the national beta field test, we identified several considerations for programs planning to implement the Feeling Heard and Understood performance measure into clinical practice, quality improvement efforts, and quality payment programs.

First, it is important to recognize that certain populations and types of visits had to be excluded from the national beta field test, but these populations and visits will be important for programs to consider as they start to use this performance measure. Pediatric populations, for instance, were not included because these patients and their families require distinct considerations for ensuring their voice is reflected in the performance measure. Telehealth visits were not included because of their low incidence before the pandemic, although the public health emergency shifted many in-person visits to a telehealth platform. Because it is likely that telehealth visits will continue in greater numbers after the pandemic ends, this performance measure should incorporate telehealth in future testing. Indeed, programs with smaller numbers of visits tended to exhibit lower reliability, which should be considered when establishing requirements for program participation.

In terms of data collection, it is important to note that scores were lower for surveys completed by telephone. Because patients (or caregivers) were only contacted as a last resort if the survey had not been completed by mail or phone, we think this effect may be due to phone respondents being disinclined to participate because of negative experiences with palliative care. Thus, future tests could explore ways to optimize data collection. The survey instrument should be used and tested in other languages as well, particularly Spanish. Although we developed a Spanish-language translation of the survey instrument used in testing, we were unable to test the reliability and validity of the performance measure using Spanish-language data collection.

While testing this performance measure, we engaged clinicians, health systems, and patients and caregivers, and we offered guidance on how to use this performance measure in practice to drive systematic quality measurement and improvement in priority areas for palliative care. Stakeholder feedback suggests that this performance measure can be successfully implemented and can provide valuable information to guide palliative care improvement in patient experience.

We also feel that future research could focus on three areas:

  • a systematic examination of survey administration modes, which may affect response rates and the representativeness of respondents
  • an evaluation of the implementation of this performance measure into clinical practice and regular use
  • an exploration and analysis of disparities in ambulatory palliative care access, utilization, and experience, including (but not limited to) racial, gender, ethnic, cultural, and language disparities.

Feeling Heard and Understood: Final Measure Specification

Performance Measure Description

Feeling Heard and Understood is a multi–data element measure consisting of four data elements:

  • Q1. “I felt heard and understood by this provider and team”
  • Q2. “I felt this provider and team put my best interests first when making recommendations about my care”
  • Q3. “I felt this provider and team saw me as a person, not just someone with a medical problem”
  • Q4. “I felt this provider and team understood what is important to me in my life.”

Performance Measure Type

This is a patient-reported outcome performance measure.


All patients ages 18 years and older who had an ambulatory palliative care visit.

Denominator Criteria

All patients ages 18 years and older on date of encounter.


Ambulatory palliative care visit, defined as

  • ICD-10 Z51.5 (Encounter for Palliative Care) OR
  • Provider Hospice and Palliative Care Specialty Code 17; AND
  • CPT 99201-99205 (New Office Visit); OR CPT 99211-99215 (Established Office Visit); or Place of Service (POS) Code 11 – Office.


An eligible provider type: physicians (including doctors of medicine, osteopathy, dental surgery, dental medicine, podiatric medicine, and optometry), osteopathic practitioners, chiropractors, physician assistants, nurse practitioners, clinical nurse specialists, certified registered nurse anesthetists, physical therapists, occupational therapists, clinical psychologists, qualified speech-language pathologists, qualified audiologists, and registered dietitians or nutrition professionals (this list is based on 2019 MIPS eligible clinician types).

Denominator Exclusions
  • Patients who do not complete at least one of the four data elements in the multi–data element measure.
  • Patients who do not complete and return the patient experience survey within six months of the eligible ambulatory palliative care visit.
  • Patients who respond on the patient experience survey that they did not receive care by the listed ambulatory palliative care provider in the past six months (i.e., disavow the provider or program).
  • Patients who were deceased when the survey reached them.
  • Patients for whom a proxy completed the entire survey on their behalf for any reason (i.e., with no patient involvement), including patients who were deceased at the time the survey reached them.


The Feeling Heard and Understood performance measure is calculated using top-box scoring. The top-box score refers to the percentage of patient respondents who give the most positive response. For each of the four data elements in this performance measure, individuals can respond using discrete values ranging from 0 (“Not at all true”) to 4 (“Completely true”). For all four data elements, the top-box numerator is the number of respondents who answer “Completely true.” An individual's score can be considered an average of the four top-box responses, and these scores are adjusted for mode of survey administration and proxy assistance. Individual scores are combined to calculate an average score for an overall palliative care program.

Missing data in the outcome are naturally accommodated among the four data elements by the modeling procedure for adjusted score estimation; no outcome imputation is necessary and should not be performed.


Ahluwalia, Sangeeta C., Brian G. Vegetabile, Maria Orlando Edelen, Claude Messan Setodji, Anthony Rodriguez, Adam Scherling, Jessica Phillips, Carrie M. Farmer, Jordan M. Harrison, Julia Bandini, Lucy Schulson, Anne Walling, Sarah Dalton, Monique Martineau, Owen Hall, Danielle Schlang, and Melissa A. Bradley, MACRA Palliative Care Quality Measure Development Testing Summary Report: Measure Name: Receiving Desired Help for Pain, Santa Monica, Calif.: RAND Corporation, RR-A400-2, 2021. As of June 1, 2022:

This research was funded by the Centers for Medicare & Medicaid Services and carried out within the Quality Measurement and Improvement Program in RAND Health Care.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.