Research Brief

Program Improvement Tools

  • The RAND Program Classification Tool allows decisionmakers to characterize and compare programs along a number of dimensions, including whether the programs have specified goals, which clinical and nonclinical areas the programs address, and whether an evaluation has been conducted.
  • The RAND Online Measure Repository helps in identifying appropriate outcome metrics for assessing program performance against specified goals and for conducting formal program outcome evaluations.
  • The RAND Program Expansion Tool offers decision support to help assess existing evidence regarding the effectiveness of an individual program and determine whether the evidence justifies continuing or expanding the program.
  • The RAND Program Manager's Guide provides practical guidance for those responsible for managing programs in how to assess and improve program performance and conduct continuous quality improvement.

Since 2001, the U.S. Department of Defense (DoD) has implemented numerous programs to support service members and their families in coping with the stressors from a decade of conflict in Iraq and Afghanistan. These programs, which address both psychological health and traumatic brain injury (TBI), number in the hundreds and vary in their size, scope, and target population. To ensure that resources are wisely invested to maximize the benefits of such programs, the Assistant Secretary of Defense for Health Affairs asked the RAND National Defense Research Institute to develop a set of tools to assist with understanding, evaluating, and improving program performance.

The four-volume toolkit is not intended to provide an overview of evaluation methods, a step-by-step guide to conducting program evaluation, or directions for conducting needs assessments. Many resources are available to support such efforts. Rather, the purpose is to help those responsible for overseeing portfolios of multiple programs to address key interrelated questions:

  • Do the programs in a given portfolio fully address the needs of the population the portfolio is intended to serve?
  • What tools are available to assess a program's effect on service members?
  • Does the evidence indicate that a particular program is ready for expansion?
  • How can program performance be improved?

In this research highlight, we discuss each of these questions in more detail and present examples of how the RAND toolkit can help with answers. Each of the RAND tools is available on the RAND website; please see the end of this document for detailed information.

The Tools

Do the Programs in a Given Portfolio Fully Address the Needs of the Population?

One of the challenges for managers who oversee a portfolio of programs is determining whether the right programs exist to address the needs of service members supported by the office overseeing the programs. This challenge is complicated by the need to understand whether there are too many, too few, or the right number of programs to meet those needs.

The RAND Program Classification Tool (R-PCT) allows portfolio managers to describe and categorize programs across a number of dimensions to better understand how existing programs may — or may not — be meeting the population's needs. This tool can be used to collect and organize information about

  • program goals
  • the types of activities conducted by the program (e.g., prevention, treatment, education)
  • how participants interact with the program (e.g., in person, on the phone)
  • the scale of the program (e.g., operating on one installation or across an entire branch of service)
  • barriers that the target population faces in using the program
  • the extent to which the program is engaged in evaluation activities
  • transferability of the program to a new location or population.

The R-PCT can be applied to a portfolio of existing programs. It can also be used to organize information regarding programs that are in the process of being developed or programs that are requesting funding. For example, a portfolio manager could compare information from the R-PCT across all the programs under his or her purview to assess whether goals are redundant across existing programs, or to identify any gaps in the existing portfolio of programs.

What Tools Are Available to Assess a Program's Effect on Service Members?

Evaluating program effectiveness is paramount for ensuring wise investment of resources available for supporting service members. The figure below helps explain where the tools fit in the process. The RAND Online Measure Repository (ROMR) is intended to support the design and implementation of evaluations that focus on program outcomes by providing an online repository that indexes and describes measures related to psychological health and TBI that have been used in both civilian and military populations.

The RAND Toolkit Provides a Set of Interrelated Resources to Help Managers Improve Program Effectiveness and Utility

The RAND Toolkit Provides a Set of Interrelated Resources to Help Managers Improve Program Effectiveness and Utility

The ROMR includes measures related to

  • psychological health, focusing on posttraumatic stress disorder, depression, anxiety, suicidal ideation, and resiliency
  • TBI, including measures of cognition, executive functioning, and memory
  • the well-being of military units, such as unit cohesion and force readiness.

The material in the ROMR is intended to help in the selection of questions and scales to be included in surveys or other self-administered tools. Instruments that are designed for clinical diagnosis are not included.

Does the Evidence Indicate That a Particular Program Is Ready for Expansion?

Once a program has been in operation for a period of time, there is often interest in expanding it to new locations or to support new populations. However, a lack of centralized program oversight might result in the proliferation of untested programs that are developed without an evidence base and that add cost and administrative inefficiencies.

The RAND Program Expansion Tool (RPET) is designed to support a centralized, systematic, and ongoing process by which DoD decisions around program support and expansion can be made based on available evidence regarding program effectiveness, the quality of evaluations conducted to assess that effectiveness, and other characteristics, such as the priority of the target population and the current policy context.

The RPET was designed for use by a decisionmaking board to systematically assess a candidate program for expansion. The tool presents a decision tree to help identify programs that are strong candidates for expansion. Within the decision tree, the quality of the evaluation and the information it provides regarding program effectiveness are prioritized to quickly identify those programs that are not candidates for expansion, or that should be ended due to lack of program effectiveness or — in rare cases — evidence of harm. Information about the priority of the population and the current policy context help to refine program maintenance or recommendations for expansion.

How Can Program Performance Be Improved?

The RAND Program Manager's Guide (RPMG) is designed as a guide for program managers and other decisionmakers to assess ongoing programs, consider options for improving program performance, implement solutions, and evaluate whether the solutions have contributed to improved program performance. The tool includes a step-by-step guide to help program managers

  • specify program goals
  • determine which aspects of performance to measure, identify appropriate measurement tools, and compare performance against goals
  • identify where problems are occurring in the program and understand why
  • identify, test, and implement possible solutions to performance problems
  • develop a process for ongoing performance monitoring.

Using the Tools

The RAND Toolkit and Its Uses

Key Question R-PCT
(Volume 1)
ROMR
(Volume 2)
RPET
(Volume 3)
RPMG
(Volume 4)
Has an evaluation been conducted? X   X  
If an evaluation has been conducted:        
Was its quality high?     X  
If the evaluation measured service members' psychological health or TBI outcomes, were the measures used in the evaluation appropriate and high quality?   X    
How effective was the program?     X  
If an evaluation has not been conducted:        
Is the program collecting outcome data that could enable an assessment of effectiveness? X      
Is the program ready for an outcome evaluation? X      
If an evaluation is being designed:        
What measures can be used to assess service members' outcomes?   X    
What approaches will best contribute to the evidence needed to make decisions regarding possible program expansion?     X  
How can the available evidence be used to determine which programs are more effective than others?     X  
How can problems affecting program effectiveness be identified?       X
How can programs improve their performance?       X

This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.