A Systematic Process to Facilitate Evidence-Informed Decisionmaking Regarding Program Expansion

Since 2001, the Department of Defense (DoD) has implemented numerous programs to support psychological health and care for traumatic brain injury (TBI) for service members and their families. A variety of factors—including increasing and ongoing awareness regarding the psychological and cognitive consequences of deployment, recommendations resulting from the work of highly visible advisory committees, the expanded numbers of mental health care providers available in military clinical care settings, and the establishment of the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury—have created significant motivation and momentum for developing such programs. These programs address various components of biological, psychological, social, spiritual, and holistic influences on psychological health along the resilience-prevention-treatment continuum and focus on a variety of clinical and nonclinical concerns.

In 2011, at the request of the Assistant Secretary of Defense for Health Affairs, RAND developed a comprehensive catalog of existing programs sponsored or funded by DoD that enhance psychological health and resilience as well as improve care and support for service members recovering from TBIs. As part of this effort, RAND identified more than 200 programs, defined as activities structured to achieve specific objectives over time that are generally driven by broad policy or clinical practice guidelines, and with specific program-related resources such as personnel, financing, and infrastructure. As part of that effort, RAND developed a series of high-level recommendations for the DoD, identifying areas where efforts may be expanded to better support the needs of service members and their families.

Despite its support of these psychological health and TBI programs, DoD lacks an approach and process to systematically develop, track, and assess the performance of this portfolio of programs. Further, there is not yet a uniform approach to decisionmaking around continued program support and expansion of particularly promising, evidence-based programs. This lack of centralized oversight may result in the proliferation of untested programs that are developed without an evidence base, but with an inefficient use of resources, added cost, and administrative inefficiencies. Further, it raises the potential that some programs—despite the best intentions of their originators—may cause harm or delay entry into the system of care and that such harm would not be identified in a timely fashion. Five specific recommendations were made to address this gap (Weinick et al., 2011):

• The evidence base regarding program effectiveness needs to be developed.
• The evidence base regarding program effectiveness needs to be centralized and made accessible across DoD.
• Programs shown to be ineffective should be discontinued and should not be replicated.
• A central authority should set overall policies and establish guidelines regarding programs, including guidelines governing the proliferation of new programs.
• Both new and existing programs should be tracked on an ongoing basis by a single entity, preferably the same organization that is charged with developing guidance regarding program proliferation.

This study describes a potential model and tools to support a centralized, systematic, and ongoing process by which decisions around program support and expansion can be made.

This study includes two tools: a Program Abstraction Form and the RAND Program Expansion Tool (RPET). The Program Abstraction Form collects relevant information from programs. In addition to collecting information on program services, population served, and demand for the program, it asks explicitly about program effectiveness (i.e., did the program result in desired outcomes?) and the design of the program evaluation used to assess program effectiveness (a poor evaluation design may lead to incorrect conclusions about the effectiveness of the program).

The RPET was designed for use by a decisionmaking board to systematically assess a candidate program for expansion, based on information provided by the program via the Program Abstraction Form. The RPET addresses four main criteria: quality of evaluation, program effectiveness, priority of the population, and priority of the policy context. Once the RPET is completed by an individual with expertise in program assessment and evaluation, RPET ratings and other program information can be shared with the board. The decisionmaking board can use the information when considering program expansion, using the included decision tree that incorporates key criteria to be considered when making such decisions. Within the decision tree, the quality of the evaluation and the information it provides regarding program effectiveness are prioritized to quickly identify those programs that are not candidates for expansion, or that should be ended due to lack of program effectiveness or—in rare cases—evidence of harm. Information about the priority of the population and the current policy context help to refine maintenance or expansion recommendations.

These tools are designed to support, not supplant, human decisionmaking, and as such do not dictate whether or how a program should be expanded. Such decisions ideally would be made by a well-rounded and representative group of individuals, or decisionmaking board, who serve as the central authority for monitoring such programs, and for decisionmaking around continual program support and potential expansion. We propose a possible model for a decisionmaking board, based on a Human Research Protection Committee, also known as an Institutional Review Board. While these types of boards and committees provide oversight for research studies, a similar model of submission to the board, triage, and full committee review for a subset of programs may provide a useful model for systematically and transparently reviewing the portfolio of DoD programs designed to enhance psychological health and resilience as well as improve care and support for service members recovering from TBIs. Through careful, consistent, and ongoing review of programs, DoD can develop a maximally effective portfolio of programs to support psychological health and services for TBI.

Reference

Weinick RM, Beckjord EB, Farmer CM, Martin LM, Gillen EM, Acosta JD, Fisher MP, Garnett J, Gonzalez GC, Helmus TC, Jaycox LH, Reynolds KA, Salcedo N, and Scharf DM, Programs Addressing Psychological Health and Traumatic Brain Injury Among U.S. Military Servicemembers and Their Families, Santa Monica, Calif.: RAND Corporation, TR-950-OSD, 2011. As of January 6, 2012:
http://www.rand.org/pubs/technical_reports/TR950.html

This research was sponsored by the the Office of the Secretary of Defense (OSD) and the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury. It was conducted in the Forces and Resources Policy Center, a RAND National Defense Research Institute (NDRI) program. NDRI is a federally funded research and development center sponsored by the OSD, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.