Evaluating Uncertainty in Cost Estimates

A cost estimate for a project, such as the development of a new aircraft or satellite system, carries with it an inherent probability that the actual cost will exceed the estimate — that certain changes that may occur over the life of an acquisition project, such as in requirements, technology, and the economic environment, will in turn change the final cost. One major approach to cost risk analysis — evaluating and quantifying the uncertainty in a cost estimate — has been to express the uncertainty as a probability distribution over a range of possible costs.

To obtain probability distributions for new and untried technologies, cost analysts have often proposed tapping the resources of experts and eliciting subjective probability distributions to quantify cost uncertainty. However, this approach has potential problems. Human beings are subject to serious biases, such as overconfidence in their ability to quantify uncertainty, which can distort their judgments. Moreover, there is little information in the professional cost-risk literature to explain how expert judgments are actually elicited. Elicitation practices are diverse — even in such disciplines as space system acquisition, which involves a limited number of organizations and is an area in which one might expect a high degree of methodological standardization.

RAND Project AIR FORCE (PAF) offers the following procedure as a starting point for cost risk analysts eliciting expert judgments regarding probability distributions for new and untried technologies:

• Use multiple independent experts.
• Ask each expert to provide, at a minimum, upper, lower, and most-likely values for the cost elements under consideration.
• Fit a simplified distribution to these three numbers, using the upper and lower values provided by each expert to bound 90 percent of the probability (when reasonable, to counteract known biases in elicitation).
• In addition to the upper, lower, and most-likely values, elicit at least two more percentiles (perhaps the 25th and 75th, as most current authors recommend).
• Provide feedback to each expert about the results of the elicitation, including the final range of nonzero probabilities, the median estimated cost, and the probability that the final cost will exceed the most-likely cost, and allow the experts to use the feedback to adjust their assessments.
• Carefully document the process and results and archive the data for future retrospective studies.

The cost-estimation community is in general agreement that probabilistic methods of quantifying and reasoning with uncertainty are the most rigorous methods of cost risk analysis. What is needed is a systematic set of empirical case studies of elicitation in cost risk analysis to allow retrospective studies of the effectiveness and accuracy of different techniques. Such case studies would provide cost risk analysts with a set of credible tools to do elicitation that can be compared and refined with further experience. Cost risk analysis is in a unique position to contribute to the development of elicitation procedures: It has a need for elicitation to quantify significant uncertainties; it has many different opportunities in government and industry to apply these techniques and test them; and it has quantitatively sophisticated practitioners who can help advance the field of elicitation.