Evaluating Uncertainty in Cost Estimates

A Practical Guide to Eliciting Expert Judgments

by Lionel A. Galway

Download Free Electronic Document

FormatFile SizeNotes
PDF file 0.1 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Brief

A cost estimate for a project, such as the development of a new aircraft or satellite system, carries with it an inherent probability that the actual cost will exceed the estimate — that certain changes that may occur over the life of an acquisition project, such as in requirements, technology, and the economic environment, will in turn change the final cost. One major approach to cost risk analysis — evaluating and quantifying the uncertainty in a cost estimate — has been to express the uncertainty as a probability distribution over a range of possible costs.

To obtain probability distributions for new and untried technologies, cost analysts have often proposed tapping the resources of experts and eliciting subjective probability distributions to quantify cost uncertainty. However, this approach has potential problems. Human beings are subject to serious biases, such as overconfidence in their ability to quantify uncertainty, which can distort their judgments. Moreover, there is little information in the professional cost-risk literature to explain how expert judgments are actually elicited. Elicitation practices are diverse — even in such disciplines as space system acquisition, which involves a limited number of organizations and is an area in which one might expect a high degree of methodological standardization.

RAND Project AIR FORCE (PAF) offers the following procedure as a starting point for cost risk analysts eliciting expert judgments regarding probability distributions for new and untried technologies:

  • Use multiple independent experts.
  • Ask each expert to provide, at a minimum, upper, lower, and most-likely values for the cost elements under consideration.
  • Fit a simplified distribution to these three numbers, using the upper and lower values provided by each expert to bound 90 percent of the probability (when reasonable, to counteract known biases in elicitation).
  • In addition to the upper, lower, and most-likely values, elicit at least two more percentiles (perhaps the 25th and 75th, as most current authors recommend).
  • Provide feedback to each expert about the results of the elicitation, including the final range of nonzero probabilities, the median estimated cost, and the probability that the final cost will exceed the most-likely cost, and allow the experts to use the feedback to adjust their assessments.
  • Carefully document the process and results and archive the data for future retrospective studies.

The cost-estimation community is in general agreement that probabilistic methods of quantifying and reasoning with uncertainty are the most rigorous methods of cost risk analysis. What is needed is a systematic set of empirical case studies of elicitation in cost risk analysis to allow retrospective studies of the effectiveness and accuracy of different techniques. Such case studies would provide cost risk analysts with a set of credible tools to do elicitation that can be compared and refined with further experience. Cost risk analysis is in a unique position to contribute to the development of elicitation procedures: It has a need for elicitation to quantify significant uncertainties; it has many different opportunities in government and industry to apply these techniques and test them; and it has quantitatively sophisticated practitioners who can help advance the field of elicitation.

Research conducted by

This report is part of the RAND Corporation Research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.