Subjective Probability Distribution Elicitation in Cost Risk Analysis: A Review
Jul 1, 2007
A Practical Guide to Eliciting Expert Judgments
|PDF file||0.1 MB||
Use Adobe Acrobat Reader version 10 or higher for the best experience.
A cost estimate for a project, such as the development of a new aircraft or satellite system, carries with it an inherent probability that the actual cost will exceed the estimate — that certain changes that may occur over the life of an acquisition project, such as in requirements, technology, and the economic environment, will in turn change the final cost. One major approach to cost risk analysis — evaluating and quantifying the uncertainty in a cost estimate — has been to express the uncertainty as a probability distribution over a range of possible costs.
To obtain probability distributions for new and untried technologies, cost analysts have often proposed tapping the resources of experts and eliciting subjective probability distributions to quantify cost uncertainty. However, this approach has potential problems. Human beings are subject to serious biases, such as overconfidence in their ability to quantify uncertainty, which can distort their judgments. Moreover, there is little information in the professional cost-risk literature to explain how expert judgments are actually elicited. Elicitation practices are diverse — even in such disciplines as space system acquisition, which involves a limited number of organizations and is an area in which one might expect a high degree of methodological standardization.
RAND Project AIR FORCE (PAF) offers the following procedure as a starting point for cost risk analysts eliciting expert judgments regarding probability distributions for new and untried technologies:
The cost-estimation community is in general agreement that probabilistic methods of quantifying and reasoning with uncertainty are the most rigorous methods of cost risk analysis. What is needed is a systematic set of empirical case studies of elicitation in cost risk analysis to allow retrospective studies of the effectiveness and accuracy of different techniques. Such case studies would provide cost risk analysts with a set of credible tools to do elicitation that can be compared and refined with further experience. Cost risk analysis is in a unique position to contribute to the development of elicitation procedures: It has a need for elicitation to quantify significant uncertainties; it has many different opportunities in government and industry to apply these techniques and test them; and it has quantitatively sophisticated practitioners who can help advance the field of elicitation.
This report is part of the RAND Corporation Research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.