Military planners and other decisionmakers often use mathematical models to simulate real-world situations for purposes such as strategic planning, needs assessment, and testing new warfighting capabilities and concepts. These models must be validated to ensure that they are an adequate representation of the real-world situation for the purposes at hand. Model validation is relatively straightforward in cases where a model is based on settled theories and reliable data. However, planners often face circumstances in which the validity of a model or its data is uncertain, even massively uncertain.
RAND Project AIR FORCE has reported a set of concepts that will improve analysts' ability to deal with uncertain models and data. Key principles include the following:
- Models should be comprehensible and explainable to subject-matter experts. One way to help validate an uncertain model is to make sure that its structure and findings are explainable in real-world terms. This goal can be accomplished by developing sound, credible, coherent, and insightful stories to illustrate a model or findings. Such stories will explain the logical relations between parts of the model, such as cause-and-effect mechanisms. Analysts should develop several stories at different levels of detail to explain the model to both fellow analysts and high-level decisionmakers.
- Models should deal effectively with uncertainty. Analysts typically seek to remove uncertainty by making assumptions about aspects of a model that are otherwise unknown. A more valid approach is to incorporate uncertainty within a model. New methods such as multiresolution, multiperspective modeling (MRMPM) and exploratory analysis can help accomplish this goal. MRMPM uses a family of models and games to approach complex problems from different points of view. This method allows analysts to cross-calibrate models that have differing resolutions, sources of data, and perspectives. Exploratory analysis exercises a model across a range of circumstances and assumptions that represent relevant uncertainties. Such analysis treats the uncertainties simultaneously rather than one at a time. This approach allows analysts to assess the robustness of conclusions to uncertain inputs.
- The concept of validation should be broadened to include appropriate qualifiers. Some models are valid for limited applications even if they cannot be applied reliably under all conditions. Under current criteria, these models would not be recognized as valid. Analysts should be able to use qualifiers such as "valid for exploratory analysis" or "valid, subject to the principal assumptions underlying the model, for exploratory analysis" when characterizing a model's validity.
This research brief summarizes the findings of Project AIR FORCE.
This report is part of the RAND Corporation Research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.