- How can Office of the Secretary of Defense-level assessors make judgments about the integration risk associated with major weapon programs for which they may not be especially familiar?
- Are there methods that can be generalized to provide a standards-based valuation of integration issues across many different types of major weapons programs?
- How can these integration risk valuation methods be developed to allow for reproducibility and traceability?
Implementing risk management principles to manage large defense acquisition programs is a priority for the U.S. defense acquisition community. To assist those decisionmakers responsible for identifying the risk associated with major weapons programs, RAND researchers developed a methodology and accompanying Excel, information-based risk tool (the "Assessor Tool"). A description of the methodology and the tool are available in a companion document, A Risk Assessment Methodology and Excel Tool for Acquisition Programs (by Lauren A. Fleishman-Mayer, Mark V. Arena, and Michael E. McMahon, RR-262-OSD, 2013). The present document is the users' manual for the Assessor Tool. The Assessor Tool offers an Office of the Secretary of Defense (OSD)-level approach to the evaluation and measurement of system integration risk. That is, it is meant for assessors, such as OSD personnel, who may not be especially familiar with the specific program under evaluation but still may need to make judgments about the program's risk. It is based on a tractable and comprehensive set of questions that can help evaluate integration risk at each point in the acquisition process. More specifically, the tool enables users to see how well integration risk is being managed by providing a standards-based valuation of integration issues that can lead to cost growth, schedule growth, and program performance. The Assessor Tool and its methodology may also be generalizable to an entire set of information-based risk assessment applications. Overall, the methodology and tool have many strengths, including being based on well-grounded theories, allowing for reproducibility and traceability, and the extensive flexibility to be used to evaluate risk for many different types of programs. To provide a benchmarking and validation of the risk scores calculated by the tool, future work could include the tool's validation by tracking its output against a program's performance.
The Assessor Tool and Its Information-Based Risk Assessment Methodology Offer an Office of the Secretary of Defense (OSD)-Level Valuation of Program Risk
- The Assessor Tool is designed for those staff involved more generally with weapon systems acquisition who need access to a systematic method of determining a program's ability to meet its goals and manage risks, and to provide one basis to report on the success of the department's compliance with system integration risk management as directed by the Weapon Systems Acquisition Reform Act.
- The risk valuation is based on established documentation that allows for a level of traceability, verifiability, and objectiveness.
The Assessor Tool Design Is Based Upon OSD-Level Assessment Questions and Utility Theory
- The methodology assumes that a set of knowledge-based standards has been developed against which to measure program risk and that a risky outcome may result if a question is not satisfied.
- The knowledge-based standards are based on the existence and completeness of Department of Defense artifacts and checklists that would be readily available to an assessor at the OSD level.
The Assessor Tool Can Be Adapted for Additional Risk-Related Assessments
- The tool and methodology are also generalizable to an entire set of information-based risk assessment applications.
- The tool could easily be tailored to insert user-determined review elements specific to previously identified technical or integration risk issues.
- Using the blank template of the tool, a user could adapt it to evaluate risk related to many other types of programs with documented knowledge-based standards in place.
- The reproducible and documented tool for integration risk assessment may be considered for program office reporting to meet Weapon Systems Acquisition Reform Act compliance as well as for other acquisition reviews and for adaptation into other program assessment tools.
- To provide a benchmarking and validation of hte risk scores calculated by the tool, future work could include the tool's validation by tracking its output against a program's performance.
This research was conducted within the Acquisition and Technology Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.
This report is part of the RAND Corporation tool series. RAND tools may include models, databases, calculators, computer code, GIS mapping tools, practitioner guidelines, web applications, and various toolkits. All RAND tools undergo rigorous peer review to ensure both high data standards and appropriate methodology in keeping with RAND's commitment to quality and objectivity.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.