Jan 8, 2014
Since 2001, the U.S. Department of Defense (DoD) has implemented numerous programs to support service members and their families in coping with the stressors from a decade of conflict in Iraq and Afghanistan. These programs, which address both psychological health and traumatic brain injury (TBI), number in the hundreds and vary in their size, scope, and target population. To ensure that resources are wisely invested to maximize the benefits of such programs, the Assistant Secretary of Defense for Health Affairs asked the RAND National Defense Research Institute to develop a set of tools to assist with understanding, evaluating, and improving program performance.
The four-volume toolkit is not intended to provide an overview of evaluation methods, a step-by-step guide to conducting program evaluation, or directions for conducting needs assessments. Many resources are available to support such efforts. Rather, the purpose is to help those responsible for overseeing portfolios of multiple programs to address key interrelated questions:
In this research highlight, we discuss each of these questions in more detail and present examples of how the RAND toolkit can help with answers. Each of the RAND tools is available on the RAND website; please see the end of this document for detailed information.
One of the challenges for managers who oversee a portfolio of programs is determining whether the right programs exist to address the needs of service members supported by the office overseeing the programs. This challenge is complicated by the need to understand whether there are too many, too few, or the right number of programs to meet those needs.
The RAND Program Classification Tool (R-PCT) allows portfolio managers to describe and categorize programs across a number of dimensions to better understand how existing programs may — or may not — be meeting the population's needs. This tool can be used to collect and organize information about
The R-PCT can be applied to a portfolio of existing programs. It can also be used to organize information regarding programs that are in the process of being developed or programs that are requesting funding. For example, a portfolio manager could compare information from the R-PCT across all the programs under his or her purview to assess whether goals are redundant across existing programs, or to identify any gaps in the existing portfolio of programs.
Evaluating program effectiveness is paramount for ensuring wise investment of resources available for supporting service members. The figure below helps explain where the tools fit in the process. The RAND Online Measure Repository (ROMR) is intended to support the design and implementation of evaluations that focus on program outcomes by providing an online repository that indexes and describes measures related to psychological health and TBI that have been used in both civilian and military populations.
The ROMR includes measures related to
The material in the ROMR is intended to help in the selection of questions and scales to be included in surveys or other self-administered tools. Instruments that are designed for clinical diagnosis are not included.
Once a program has been in operation for a period of time, there is often interest in expanding it to new locations or to support new populations. However, a lack of centralized program oversight might result in the proliferation of untested programs that are developed without an evidence base and that add cost and administrative inefficiencies.
The RAND Program Expansion Tool (RPET) is designed to support a centralized, systematic, and ongoing process by which DoD decisions around program support and expansion can be made based on available evidence regarding program effectiveness, the quality of evaluations conducted to assess that effectiveness, and other characteristics, such as the priority of the target population and the current policy context.
The RPET was designed for use by a decisionmaking board to systematically assess a candidate program for expansion. The tool presents a decision tree to help identify programs that are strong candidates for expansion. Within the decision tree, the quality of the evaluation and the information it provides regarding program effectiveness are prioritized to quickly identify those programs that are not candidates for expansion, or that should be ended due to lack of program effectiveness or — in rare cases — evidence of harm. Information about the priority of the population and the current policy context help to refine program maintenance or recommendations for expansion.
The RAND Program Manager's Guide (RPMG) is designed as a guide for program managers and other decisionmakers to assess ongoing programs, consider options for improving program performance, implement solutions, and evaluate whether the solutions have contributed to improved program performance. The tool includes a step-by-step guide to help program managers
|Has an evaluation been conducted?
|If an evaluation has been conducted:
|Was its quality high?
|If the evaluation measured service members' psychological health or TBI outcomes, were the measures used in the evaluation appropriate and high quality?
|How effective was the program?
|If an evaluation has not been conducted:
|Is the program collecting outcome data that could enable an assessment of effectiveness?
|Is the program ready for an outcome evaluation?
|If an evaluation is being designed:
|What measures can be used to assess service members' outcomes?
|What approaches will best contribute to the evidence needed to make decisions regarding possible program expansion?
|How can the available evidence be used to determine which programs are more effective than others?
|How can problems affecting program effectiveness be identified?
|How can programs improve their performance?