Better Methods of Analyzing Cost Uncertainty Can Improve Acquisition Decisionmaking

Cost estimation is a process that attempts to forecast the future expenditures for some capital asset, hardware, service, or capability. Despite being a highly quantitative field, cost estimation and the values it predicts are uncertain. An estimate is a possible or likely outcome, but not necessarily the outcome that will actually transpire. This uncertainty arises because estimators do not have perfect information about future events and the validity of assumptions that underpin an estimate. Uncertainty may result from an absence of critical technical information, the presence of new technologies or approaches that do not have historical analogues for comparison, the evolution of requirements, or changes in economic conditions.

The Office of the Secretary of Defense and the military departments have historically underestimated and underfunded the cost of buying new weapon systems (e.g., by more than 40 percent at Milestone II). Much of this cost growth is thought to be the result of unforeseen (but knowable) circumstances when the estimate was developed. In the interest of generating more informative cost estimates, the Air Force Cost Analysis Agency and the Air Force cost analysis community want to formulate and implement a cost uncertainty analysis policy.

To help support this effort, RAND Project AIR FORCE (PAF) studied a variety of cost uncertainty assessment methodologies, examined how these methods and policies relate to a total portfolio of programs, and explored how risk information can be communicated to senior decisionmakers in a clear and understandable way. PAF recommends that any cost uncertainty analysis policy reflect the following:

• A single uncertainty analysis method should not be stipulated for all circumstances and programs. It is not practical to prefer one specific cost uncertainty analysis methodology in all cases. Rather, the policy should offer the flexibility to use different assessment methods. These appropriate methods fall into three classes: historical, sensitivity, and probabilistic. Moreover, a combination of methods might be desirable and more effective in communicating risks to decisionmakers.
• A uniform communications format should be used. PAF suggests a basic three-point format consisting of low, base, and high values as a minimum basis for displaying risk analysis. The advantages of such a format are that it is independent of the method employed and that it can be easily communicated to decisonmakers.
• A record of cost estimate accuracy should be tracked and updated periodically. Comparing estimates with final costs will enable organizations to identify areas where they may have difficulty estimating and sources of uncertainty that were not adequately examined.
• Risk reserves should be an accepted acquisition and funding practice. Establishing reserves to cover unforeseen costs will involve a cultural change within the Department of Defense and Congress. The current approach of burying a reserve within the elements of the estimate makes it difficult to do a retrospective analysis of whether the appropriate level of reserve was set, and to move reserves, when needed, between elements of a large program.

Effective cost uncertainty analysis will help decisionmakers understand the nature of potential risk and funding exposure and will aid in the development of more realistic cost estimates by critically evaluating program assumptions and identifying technical issues.