Probabilistic Forecasts and Reproducing Scoring Systems

Discusses ways to exploit in probabilistic terms the judgment of experts on political, economic, or military problems — frequently the best information available. The Memorandum considers ways to structure an incentive system in order to elicit the best possible probabilistic forecasts and touches on methods for combining several into consensus forecasts. Personal estimates, inherently uncertain, should be couched in probabilistic terms. Such an approach provides a concise expression of subjective uncertainty, an operational self-rating of confidence in the forecast, and readily usable data for decision-theoretic models and for combination with similar forecasts. "Reproducing scoring systems" are those free of incentives to distort by exaggeration or understatement. In political, economic, and military forecasts, however, these systems may not encourage forecasters to maximize their expected gains, may be polluted by conflict of interest, and may require many forecasts to distinguish the accurate from the inaccurate forecasters. (See also RM-5888, RM-6115, RM-6118.)