Measuring the Quality of Care for Psychological Health Conditions in the Military Health System

In recent years, the number of U.S. service members treated for psychological health (PH) conditions has increased substantially. In particular, at least two PH conditions, posttraumatic stress disorder (PTSD) and major depressive disorder (MDD), have prevalence estimates ranging from 4 to 20 percent for PTSD and 5 to 37 percent for MDD (Institute of Medicine [IOM], 2013, p. 21; Ramchand, Schell, et al., 2010; Schell and Marshall, 2008). Delivering quality care to service members with these conditions is a high-priority goal for the military health system (MHS). Meeting this goal requires understanding the extent to which the care the MHS provides is consistent with evidence-based clinical practice guidelines and its own standards for quality. However, there is currently no MHS-wide system in place to evaluate quality of care for PH conditions, to assess whether the care is improving patient outcomes, or to identify potential areas for improvement.

Purpose and Approach

In order to better understand these issues, the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) asked a team of researchers from the RAND Corporation to develop a framework to identify and classify a set of measures for monitoring the quality of care provided by the MHS for two prevalent PH conditions: PTSD and MDD. The goal of this project was to identify, develop, and describe a set of candidate quality measures to assess care for PTSD and MDD.

To accomplish this goal, we performed two tasks:

• We developed a conceptual framework for assessing the quality of care for PH conditions.
• We identified a candidate set of measures for monitoring, assessing, and improving the quality of care for two high-priority PH conditions in the MHS: PTSD and MDD.

This article describes our research approach and the candidate measure sets for PTSD and MDD that we identified. The current task did not include implementation planning but rather was designed to provide the foundation for future DCoE–RAND work to pilot and later implement a subset of these measures.

How Quality Measures Are Used

Quality measures, also called performance measures, indicators, or metrics, provide a way to measure how well health care is being delivered. Such measures provide information about the health care system and highlight areas in which providers can take action to make health care more effective, safe, efficient, and equitable (National Quality Forum [NQF], undated). Quality measure scores are typically presented as the percentage of eligible patients who received recommended care (e.g., percentage of new PTSD patients screened for depression).

According to the Agency for Healthcare Research and Quality (AHRQ) (undated), organizations generally use quality measures for one or more of these three purposes:

• quality improvement (QI). Health care organizations can focus on internal or external QI. Internal QI programs measure the quality of care within a health care organization or system. External QI programs measure quality in several health care organizations and compare performance across those organizations (AHRQ, undated).
• accountability. An organization can use measures for accountability in any of three ways: public reporting, performance-based payment, and professional certification.
• research. Quality measures are used in research studies to measure how frequently evidence-based care is provided to patients and whether care differs across patient subgroups and health care settings, as well as to measure the patient response to interventions.

We illustrate the impact of quality measurement with an example of how care for mental health conditions in commercial health plans has improved steadily for more than a decade, based on two Healthcare Effectiveness Data and Information Set (HEDIS®) quality measures (National Committee for Quality Assurance, 2013).

Results

Framework for Identifying Candidate Measures

The RAND team conducted a review of existing quality measures and identified 530 measures relevant to PTSD, MDD, or other PH conditions. A group of RAND clinicians and experts in mental health quality measurement reviewed the comprehensive list of quality measures and selected candidate measures based on the extent to which each of the following was true:

• Adequate scientific evidence or professional consensus exists to support a link between the performance of care specified by the measure and health benefits to patients with PTSD or MDD. This included support from existing clinical practice guidelines, either from a professional organization (e.g., American Psychiatric Association) or U.S. Department of Veterans Affairs (VA) and U.S. Department of Defense (DoD) (validity).
• We would consider a provider or organization with significantly higher rates of adherence to the measure a higher-quality provider or organization (validity).
• The measure had been previously applied to military populations or veterans (feasibility of MHS implementation).
• Either the measure applies to many individuals or there would be serious adverse consequences from not adhering to the indicator (importance).
• There is variation among providers in the provision of the specified care indicating a quality gap and room for performance improvement (importance).
• The information necessary to determine adherence within the MHS is likely to be available in administrative data, in the medical record, or via patient (or family) interview (feasibility of MHS implementation).
• NQF had endorsed the measure. We used that endorsement as a proxy for importance, validity, reliability, feasibility, and usability.

Team members assessed the measures individually by providing a global rating that weighed these criteria, and then the group convened to discuss global ratings and identify the candidate measure sets. In the group discussion, we placed particular emphasis on identifying measures that could be targets for further refinement and field-testing within military treatment facilities (MTFs). DCoE and RAND will complete field-testing the measures during ongoing efforts. We will select measures for field-testing according to (1) high-priority areas, which DCoE will determine, and (2) the feasibility of implementing the measure using existing data sources.

We developed a two-dimensional framework as a tool for gap analysis—that is, to assist us in evaluating whether the existing measures from the environmental scan address important aspects of care for a particular PH condition. We considered multiple dimensions and decided on two. The first dimension relates to the continuum of care, which includes five phases that the DCoE portfolio of research has highlighted (as shown in Figure 1): prevention, screening, assessment, treatment, and reintegration.

The second dimension is the type of measure. Figure 2 shows the five types, which we adapted from the Donabedian model of health care quality (Donabedian, 2003). Structure measures assess the settings and available resources in which patients receive health care and the capacity to provide care. Process measures assess whether a recommended care process or event takes place during care or the degree to which a procedure or treatment is provided in a manner that reflects fidelity to the evidence base supporting its use. Outcome measures describe the outcome of care in terms of patient improvement, recovery, restoration of function, avoidance of deterioration, or survival. Patient experience measures assess patients' perceptions of their providers, the care patients receive, and their health outcomes. Resource use measures relate to the resources expended during the care for a patient with a particular condition.

Using these two dimensions, we created a matrix for characterizing the candidate measures (see Table 1). This matrix serves to highlight areas in which promising candidate measures exist, as well as areas in which measures are sparse or missing completely.

Measure Type	Care Continuum										Total
	Prevention		Screening		Assessment		Treatment		Reintegration
	PTSD	MDD	PTSD	MDD	PTSD	MDD	PTSD	MDD	PTSD	MDD	PTSD	MDD
Structure	0	0	0	0	0	0	2	2	0	0	2	2
Process	0	0	1	1	4	4	9	9	0	0	14	14
Outcome	0	0	0	0	0	0	5	5	0	0	5	5
Patient experience											6	6
Resource use											2	2
Total											29	29

Candidate Measure Sets

We identified 58 candidate quality measures—29 each for PTSD and MDD—and placed these across the care continuum. By design, to provide adequate choice during ongoing efforts, which will include evaluation of measure feasibility, utility, and functionality across MHS settings, we selected a large set of candidate measures. Ten measures in the PTSD set are PTSD-specific, and ten measures in the MDD set are MDD-specific. Nineteen of the measures in each set could apply to care for patients with a range of PH conditions, including PTSD and MDD. As shown in Table 1, these measures clustered heavily in the treatment phase for both conditions. The most common measures were process measures. The measures most commonly appropriate for use across the entire care continuum were either measures of patient experience or resource use.

Examination of Need for New Measures

Our analysis identified two care continuum phases in which there are no existing measures: prevention and reintegration. For the prevention phase of the care continuum, we noted that providers in primary care and specialty mental health care are rarely tasked with mental health prevention. We therefore chose to prioritize components of quality care that are more often the responsibility primarily of providers operating in a treatment setting. For the reintegration phase, we considered but did not select two reintegration measures because it may be particularly challenging for these measures to accurately capture the effect of appropriate reintegration practices rather than differences in baseline severity or time away from duty to receive appropriate treatment. Finally, we noted that there was no existing structure measure to assess the availability of inpatient care and created a new measure to assess inpatient availability.

Our goal was to develop quality measures to assess care provided within MHS treatment settings (e.g., primary care, specialty mental health care). These treatment settings are not tasked with prevention of mental health conditions, which usually occurs at a population level.* Typically, service members accessing treatment have already developed PH symptoms. However, the health care system could implement some best practices in prevention (e.g., avoiding immediate mental health services for trauma-exposed individuals without acute stress symptoms) (Roberts et al., 2009; Rose et al., 2002). Our belief is that such an event would occur rarely at the level of care for which we are providing recommendations and therefore is not an initial priority area. Instead, we focus on components of quality care that are the responsibility primarily of providers operating in a treatment setting. As administrators and policymakers seek to expand the scope of quality assessment, measurement of the quality of prevention services may become a priority.

Study Limitations

This article describes a review of quality measures to identify high-priority measure sets to assess care for PTSD and MDD. Our approach to developing these measure sets has some limitations. First, we relied on an internal team of RAND experts in mental health and quality measurement to select candidate measures from among multiple measures representing the same measure concept. As part of this process, the experts prioritized measures primarily based on retaining measures that were developed and tested using rigorous methods. In some cases, they also considered whether there was adequate scientific evidence or professional consensus to support a link between the care specified by the measure and the health benefits to patients with PTSD or depression. Although it may have been improved by a formal expert panel process, this process does not require a formal expert panel. Second, we focused on identifying high-priority measure sets that would be relevant for service members; additional measures could be useful to assess care for retirees and family members. Third, complete technical specifications were not readily available for each of the 530 measures, so the review of existing measures focused on available information, the level of detail of which varied across measures. Another limitation of the study is that clinical practice guidelines typically provide general guidance and, even when tailored to individual clinical characteristics, cannot anticipate individual patient preferences (Montori, Brito, and Murad, 2013). Moreover, when developing measures, most developers' accessible data sources do not reliably contain systematic information about patient preferences related to guideline-based recommendations. This limitation applies to all quality measures. Although some efforts have tried to incorporate patient preferences into quality measures (e.g., measures of goal setting and attainment in setting a treatment plan), we rarely identified such measures specifically targeted toward MDD and PTSD.

In addition, although we relied on one framework for categorizing measures, consideration of other frameworks in future work could be helpful. Under the NQF project on Patient Outcomes Measures (NQF, 2011a), a group of mental health experts developed a measurement framework for mental health and substance use. The framework encompasses five characteristics the steering committee considered to be important aspects of measuring the quality of mental health care: inclusion of mental health in broad, cross-cutting measures; consumer, patient, family, and caregiver satisfaction; promotion of healthy behaviors and environment; nontraditional measures (e.g., homelessness); and accountability and care coordination. Alternatively, the IOM aims (IOM, 2001) for improving quality of care—safety, effectiveness, patient-centeredness, timeliness, efficiency, and equitability—might be utilized to assess the completeness of the current set of candidate measures and the direction for future expansion.

Also, the measures have not been specifically assessed for adaptability and feasibility across the range of MHS settings. This article is not intended as a guide for implementing and using the listed quality measures but rather describes those foundational first steps that must be completed prior to doing so. The implementation of quality measures is a complex and separate process, and many contextual factors, such as costs, organizational policy and culture, readiness for implementation, and the availability of relevant data influence it (Damschroder et al., 2009; Nicholas et al., 2001). Planning for implementation also includes identification of senior and local clinical champions, solicitation of staff and provider input, provider education, pilot-testing, and evaluation. Developing an implementation strategy will be part of future DCoE–RAND efforts. Once a plan is developed and measures for implementation are selected, the detailed technical specifications to define the application of those measures in that particular setting would be defined while maintaining the basic definitional integrity of the measures as described here.

Recommendations

We offer three recommendations for further work with respect to measures in support of DoD's continuing efforts to improve quality of care for PH conditions:

Recommendation 1: Select a subset of high-priority, feasible quality measures to pilot-test and implement. This study includes 58 candidate measures to assess care for PTSD and MDD. We do not expect that it will be appropriate or feasible to implement all 58 of these measures. Instead, we present a candidate set of measures to consider for future implementation. For each measure, we have estimated the feasibility, based on our assessment of currently available data sources, of using the measure to assess quality of care. In addition, we acknowledge variability in how care is structured and delivered across service branches and that the delivery of PH care within DoD is constantly evolving, which will affect which measures DoD deems high priority. DoD should consider focusing initial quality measurement efforts on feasible measures that assess high-priority aspects of care at the time of implementation.

Recommendation 2: Consider structured documentation in the medical record to facilitate capturing the data necessary to compute quality measures. Many candidate quality measures rely on important clinical details that are typically documented in the medical record. Although quality measures based on administrative data are typically more feasible to implement (e.g., whether a patient received the recommended number of psychotherapy visits), these data typically do not capture many important aspects of the process of care (e.g., whether the psychotherapy delivered was evidence-based). Integrating structured documentation into the medical record is one method to potentially increase the breadth and value of data that systems routinely capture. The Veterans Health Administration (VHA) is currently pilot-testing these kinds of structured chart notes (Karlin and Cross, 2014). In addition, DCoE is currently developing an Alternative Input Method (AIM) form to capture adherence to a PTSD clinical pathway. Although structured documentation increases feasibility of measure implementation and potential data capture, it has important limitations. It is currently unknown how clinicians use structured documentation and whether these approaches accurately capture the care delivered or are tightly related to clinical outcomes. Natural-language processing (NLP) is another emerging technology that seeks to provide a means for higher-quality data collection without manual medical record review (Shiner et al., 2012). Pilot-testing and evaluating new approaches to structured documentation and data retrieval are important areas of future research.

Recommendation 3: Develop a process for ongoing assessment of quality of care. A single effort to assess quality of care will be of limited use. Instead, assessing the quality of care for PH conditions should occur as part of ongoing QI efforts, through a continuous feedback loop. As DoD continues to pursue strategies to assess the quality of care, it is essential to consider how these efforts will be sustained and what resources will be required to support ongoing quality measurement.

References

Agency for Healthcare Research and Quality, “Uses of Quality Measures,” undated; referenced 2013. As of August 23, 2013:
http://www.qualitymeasures.ahrq.gov/selecting-and-using/using.aspx

AHRQ—See Agency for Healthcare Research and Quality.

Damschroder, Laura J., David C. Aron, Rosalind E. Keith, Susan R. Kirsh, Jeffery A. Alexander, and Julie C. Lowery, “Fostering Implementation of Health Services Research Findings into Practice: A Consolidated Framework for Advancing Implementation Science,” Implementation Science, Vol. 4, No. 1, August 7, 2009. As of January 30, 2014:
http://www.implementationscience.com/content/pdf/1748-5908-4-50.pdf

Donabedian, Avedis, An Introduction to Quality Assurance in Health Care, New York: Oxford University Press, 2003.

Institute of Medicine, Returning Home from Iraq and Afghanistan: Assessment of Readjustment Needs of Veterans, Service Members, and Their Families, Washington, D.C.: National Academies Press, 2013. As of May 9, 2014:
http://www.nap.edu/catalog.php?record_id=13499

IOM—See Institute of Medicine.

Karlin, Bradley E., and Gerald Cross, “From the Laboratory to the Therapy Room: National Dissemination and Implementation of Evidence-Based Psychotherapies in the U.S. Department of Veterans Affairs Health Care System,” American Psychologist, Vol. 69, No. 1, January 2014, pp. 19–33.

Montori, Victor M., Juan Pablo Brito, and M. Hassan Murad, “The Optimal Practice of Evidence-Based Medicine: Incorporating Patient Preferences in Practice Guidelines,” JAMA, Vol. 310, No. 23, December 18, 2013, pp. 2503–2504.

National Committee for Quality Assurance, Improving Quality and Patient Experience: The State of Health Care Quality 2013, 2013. As of December 20, 2014:
http://www.ncqa.org/Portals/0/Newsroom/SOHC/2013/SOHC-web_version_report.pdf

National Quality Forum [NQF], “Who We Are,” undated; referenced September 10, 2013.

NQF—See National Quality Forum.

Nicholas, Will, Donna O. Farley, Mary E. Vaiana, and Shan Cretin, Putting Practice Guidelines to Work in the Department of Defense Medical System: A Guide for Action, Santa Monica, Calif.: RAND Corporation, MR-1267-A, 2001. As of May 12, 2014:
http://www.rand.org/content/rand/pubs/monograph_reports/MR1267.html

Ramchand, Rajeev, Terry L. Schell, Benjamin R. Karney, Karen Chan Osilla, Rachel M. Burns, and Leah Barnes Caldarone, “Disparate Prevalence Estimates of PTSD Among Service Members Who Served in Iraq and Afghanistan: Possible Explanations,” Journal of Traumatic Stress, Vol. 23, No. 1, February 2010, pp. 59–68.

Roberts, Neil P., Neil J. Kitchiner, Justin Kenardy, and Jonathan Bisson, “Multiple Session Early Psychological Interventions for the Prevention of Post-Traumatic Stress Disorder,” Cochrane Database of Systematic Reviews, Vol. 3, July 8, 2009, p. CD006869.

Rose, Suzanna C., Jonathan Bisson, Rachel Churchill, and Simon Wessely, “Psychological Debriefing for Preventing Post Traumatic Stress Disorder (PTSD),” Cochrane Database of Systematic Reviews, Vol. 2, No. 2, 2002, p. CD000560.

Schell, Terry L., and Grant N. Marshall, “Survey of Individuals Previously Deployed for OEF/OIF,” in Terri Tanielian and Lisa H. Jaycox, eds., Invisible Wounds of War: Psychological and Cognitive Injuries, Their Consequences, and Services to Assist Recovery, Santa Monica, Calif.: RAND Corporation, MG-720-CCF, 2008, pp. 87–116. As of May 12, 2014:
http://www.rand.org/pubs/monographs/MG720.html

Shiner, Brian, Leonard W. D'Avolio, Thien M. Nguyen, Maha H. Zayed, Bradley V. Watts, and Louis Fiore, “Automated Classification of Psychotherapy Note Text: Implications for Quality Assessment in PTSD Care,” Journal of Evaluation in Clinical Practice, Vol. 18, No. 3, June 2012, pp. 698–701.

Note

* Indeed, we acknowledge that major efforts are under way across DoD to address prevention of mental health problems by building resilience. However, developing measures to assess these efforts was beyond the scope of this project.

This research was sponsored by the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) and conducted within the Forces and Resources Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.