Improving Programs that Address Psychological Health and Traumatic Brain Injury

Since 2001, the U.S. Department of Defense (DoD) has implemented numerous programs to support service members and their families in coping with the stressors from a decade of conflict in Iraq and Afghanistan. These programs, which address both psychological health and traumatic brain injury (TBI), number in the hundreds and vary in their size, scope, and target population. To ensure that resources are wisely invested to maximize the benefits of such programs, the Assistant Secretary of Defense for Health Affairs asked the RAND National Defense Research Institute to develop a set of tools to assist with understanding, evaluating, and improving program performance.

The four-volume toolkit is not intended to provide an overview of evaluation methods, a step-by-step guide to conducting program evaluation, or directions for conducting needs assessments. Many resources are available to support such efforts. Rather, the purpose is to help those responsible for overseeing portfolios of multiple programs to address key interrelated questions:

• Do the programs in a given portfolio fully address the needs of the population the portfolio is intended to serve?
• What tools are available to assess a program's effect on service members?
• Does the evidence indicate that a particular program is ready for expansion?
• How can program performance be improved?

In this research highlight, we discuss each of these questions in more detail and present examples of how the RAND toolkit can help with answers. Each of the RAND tools is available on the RAND website; please see the end of this document for detailed information.

The Tools

Do the Programs in a Given Portfolio Fully Address the Needs of the Population?

One of the challenges for managers who oversee a portfolio of programs is determining whether the right programs exist to address the needs of service members supported by the office overseeing the programs. This challenge is complicated by the need to understand whether there are too many, too few, or the right number of programs to meet those needs.

The RAND Program Classification Tool (R-PCT) allows portfolio managers to describe and categorize programs across a number of dimensions to better understand how existing programs may — or may not — be meeting the population's needs. This tool can be used to collect and organize information about

• program goals
• the types of activities conducted by the program (e.g., prevention, treatment, education)
• how participants interact with the program (e.g., in person, on the phone)
• the scale of the program (e.g., operating on one installation or across an entire branch of service)
• barriers that the target population faces in using the program
• the extent to which the program is engaged in evaluation activities
• transferability of the program to a new location or population.

The R-PCT can be applied to a portfolio of existing programs. It can also be used to organize information regarding programs that are in the process of being developed or programs that are requesting funding. For example, a portfolio manager could compare information from the R-PCT across all the programs under his or her purview to assess whether goals are redundant across existing programs, or to identify any gaps in the existing portfolio of programs.

What Tools Are Available to Assess a Program's Effect on Service Members?

Evaluating program effectiveness is paramount for ensuring wise investment of resources available for supporting service members. The figure below helps explain where the tools fit in the process. The RAND Online Measure Repository (ROMR) is intended to support the design and implementation of evaluations that focus on program outcomes by providing an online repository that indexes and describes measures related to psychological health and TBI that have been used in both civilian and military populations.

The ROMR includes measures related to

• psychological health, focusing on posttraumatic stress disorder, depression, anxiety, suicidal ideation, and resiliency
• TBI, including measures of cognition, executive functioning, and memory
• the well-being of military units, such as unit cohesion and force readiness.

The material in the ROMR is intended to help in the selection of questions and scales to be included in surveys or other self-administered tools. Instruments that are designed for clinical diagnosis are not included.

Does the Evidence Indicate That a Particular Program Is Ready for Expansion?

Once a program has been in operation for a period of time, there is often interest in expanding it to new locations or to support new populations. However, a lack of centralized program oversight might result in the proliferation of untested programs that are developed without an evidence base and that add cost and administrative inefficiencies.

The RAND Program Expansion Tool (RPET) is designed to support a centralized, systematic, and ongoing process by which DoD decisions around program support and expansion can be made based on available evidence regarding program effectiveness, the quality of evaluations conducted to assess that effectiveness, and other characteristics, such as the priority of the target population and the current policy context.

The RPET was designed for use by a decisionmaking board to systematically assess a candidate program for expansion. The tool presents a decision tree to help identify programs that are strong candidates for expansion. Within the decision tree, the quality of the evaluation and the information it provides regarding program effectiveness are prioritized to quickly identify those programs that are not candidates for expansion, or that should be ended due to lack of program effectiveness or — in rare cases — evidence of harm. Information about the priority of the population and the current policy context help to refine program maintenance or recommendations for expansion.

How Can Program Performance Be Improved?

The RAND Program Manager's Guide (RPMG) is designed as a guide for program managers and other decisionmakers to assess ongoing programs, consider options for improving program performance, implement solutions, and evaluate whether the solutions have contributed to improved program performance. The tool includes a step-by-step guide to help program managers

• specify program goals
• determine which aspects of performance to measure, identify appropriate measurement tools, and compare performance against goals
• identify where problems are occurring in the program and understand why
• identify, test, and implement possible solutions to performance problems
• develop a process for ongoing performance monitoring.

Using the Tools

The RAND Toolkit and Its Uses