To meet the growing need for services to support psychological health and care for traumatic brain injury (TBI), the Department of Defense (DoD) has developed and implemented a wide range of programs in recent years (Weinick et al., 2011). Given limited resources and the considerable investments that have been made in developing these programs, it is critical to ensure that programs are operating as effectively and efficiently as possible. To do this, program performance must be assessed on a regular basis and programs must continuously seek to optimize and improve performance. Without knowing the areas where a program may be falling short, it is not possible to ensure that the program is delivering the best possible services and operating as effectively and efficiently as possible.
This study describes a tool intended as a guide for program managers and others who seek to assess the performance of ongoing programs and improve their quality. In presenting this tool, we are mindful of the realities of program improvement and assessment activities in military settings. Because of military command structures, decisions about the need to conduct program improvement activities may not reside with the individual who created the program or the individual charged with managing it. Furthermore, individuals responsible for managing the program may not necessarily control how the program is implemented. Because of these realities, we adopted an approach to program improvement in this guide that can be implemented by individuals with varying degrees of control over the program that they manage. In addition, since there is wide variation in the types of psychological health and TBI programs conducted and/or funded by DoD, we chose to keep the tool focused generally on program improvement rather than attempting to address specific elements of the DoD programs in this area. We felt that this general approach would be more useful to the range of potential users, who can then adapt this approach to program-specific conditions.
The tool is organized around a series of key questions about program improvement in general. The questions can be adapted to specific DoD psychological health or TBI programs as needed.
The tool's key questions include the following:
- Is the program accomplishing its intended goals? The first step in assessing program performance is to determine whether the program is working well. This step involves identifying whether the program has clearly defined goals, articulating those goals as clearly as possible (or defining them if none can be identified), and determining how best to measure the program's performance in reaching its goals.
- If the program is not accomplishing its intended goals, where are problems arising? If the program is not working as well as expected, the next step is to pinpoint as specifically as possible what is wrong. Often, this step involves describing clearly how the program is supposed to work, identifying the key program positions and the players who fill them, and tracing the activities of these key players across the program's operations.
- What are potential solutions for addressing the problems and what can guide the selection of which ones to implement? Once the problems have been pinpointed, a range of potential solutions can be identified and considered according to their feasibility and their likelihood of addressing the problem. Potential solutions can come from many different sources. They are found in the professional literature, suggested by people who may be interviewed, or derived from common sense and past experiences. The source of the solutions is less relevant; what is important is considering as wide and as comprehensive a range of potential solutions as possible, regardless of whether they seem feasible when first considered. Selecting which solutions to implement involves rating the proposed solutions by potential effectiveness, cost, and feasibility, and then determining the best solution through discussions with team members and other stakeholders.
- How can solutions be implemented? Implementing a solution or set of solutions typically involves multiple steps, such as developing an implementation plan, informing people about coming changes, ensuring that people understand why the changes are needed, and making people in the chain of command aware of the implementation plan. In addition, implementing most changes means that at least some people will need to change how they think and what they do.
- How well are the solutions as implemented addressing the problem? Because many programs may have limited available time or resources, the most practical method for assessing whether the changes have improved program performance is to follow a relatively simple approach known as “Plan—Do—Study—Act.” This approach could also be known as “try it and see if it works.”
- How can a program be monitored to ensure continued success? Once the specific problems identified have been addressed and the program is performing at its expected level, it is important to continue monitoring performance to ensure that the program remains free of those problems and to seek opportunities for further improvement. With periodic checks and ongoing monitoring, it is possible to identify early warning signs that program performance may be declining.
It is important to note that this study provides a tool to assess the performance of individual programs and is not intended for comparisons across programs to determine their relative effectiveness. In addition, this study does not provide the information necessary to determine whether a particular program should be ended, or whether additional resources should be devoted to improve the program's performance. Rather, this study enables program managers and others to assess whether the program is meeting its goals and, if not, to develop a plan for identifying and solving problems that are hampering performance.
Reference
Weinick RM, Beckjord EB, Farmer CM, Martin LT, Gillen EM, Acosta J, Fisher MP, Garnett J, Gonzalez GC, Helmus TC, Jaycox LH, Reynolds K, Salcedo N, and Scharf DM, Programs Addressing Psychological Health and Traumatic Brain Injury Among U.S. Military Servicemembers and Their Families, Santa Monica, Calif.: RAND Corporation, TR-950-OSD, 2011. As of October 8, 2013:
http://www.rand.org/pubs/technical_reports/TR950.html
This research was sponsored by the the Office of the Secretary of Defense (OSD) and the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury. It was conducted in the Forces and Resources Policy Center, a RAND National Defense Research Institute (NDRI) program. NDRI is a federally funded research and development center sponsored by the OSD, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.
