Military planners and other decisionmakers often use mathematical models to simulate real-world situations for purposes such as strategic planning, needs assessment, and testing new warfighting capabilities and concepts. These models must be validated to ensure that they are an adequate representation of the real-world situation for the purposes at hand. Model validation is relatively straightforward in cases where a model is based on settled theories and reliable data. However, planners often face circumstances in which the validity of a model or its data is uncertain, even massively uncertain.
RAND Project AIR FORCE has reported a set of concepts that will improve analysts' ability to deal with uncertain models and data. Key principles include the following:
- Models should be comprehensible and explainable to subject-matter experts. One way to help validate an uncertain model is to make sure that its structure and findings are explainable in real-world terms. This goal can be accomplished by developing sound, credible, coherent, and insightful stories to illustrate a model or findings. Such stories will explain the logical relations between parts of the model, such as cause-and-effect mechanisms. Analysts should develop several stories at different levels of detail to explain the model to both fellow analysts and high-level decisionmakers.
- Models should deal effectively with uncertainty. Analysts typically seek to remove uncertainty by making assumptions about aspects of a model that are otherwise unknown. A more valid approach is to incorporate uncertainty within a model. New methods such as multiresolution, multiperspective modeling (MRMPM) and exploratory analysis can help accomplish this goal. MRMPM uses a family of models and games to approach complex problems from different points of view. This method allows analysts to cross-calibrate models that have differing resolutions, sources of data, and perspectives. Exploratory analysis exercises a model across a range of circumstances and assumptions that represent relevant uncertainties. Such analysis treats the uncertainties simultaneously rather than one at a time. This approach allows analysts to assess the robustness of conclusions to uncertain inputs.
- The concept of validation should be broadened to include appropriate qualifiers. Some models are valid for limited applications even if they cannot be applied reliably under all conditions. Under current criteria, these models would not be recognized as valid. Analysts should be able to use qualifiers such as "valid for exploratory analysis" or "valid, subject to the principal assumptions underlying the model, for exploratory analysis" when characterizing a model's validity.
This research brief summarizes the findings of Project AIR FORCE.
This report is part of the RAND Corporation research brief series. RAND research briefs present policy-oriented summaries of individual published, peer-reviewed documents or of a body of published work.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.