The decision analysis community faces obstacles in moving new methods, tools, and paradigms from theory to practice, partly because of the difficulties in demonstrating value proposition. In this report, RAND researchers use a structured comparison game to examine the value proposition of different analytic inputs (scenario-based versus Robust Decision Making) on a sample U.S. Department of Defense decision about force structure.
- Can a structured comparison game provide evidence about the relative value of different analytic products?
- In the tabletop game conducted, did teams make different decisions when presented with scenario-based versus RDM analysis? How did the decisionmaking process and resulting decisions differ?
The decision analysis community faces obstacles in moving new methods, tools, and paradigms from the academy to the boardroom or the White House Situation Room. Change requires investments of time and resources that are hard to justify if one cannot show the value proposition of an innovative approach — that is, that it truly improves decisionmaking processes and the resulting decisions. However, without an application to evaluate, such evidence may be in short supply, and the approach may therefore not be adopted. In this report, RAND researchers examine that value proposition by comparing the effects of two different analytic inputs on a sample U.S. Department of Defense decision about future force structure. The researchers designed a structured comparison game in which groups of mid-level and senior players simulating defense officials were briefed with two different types of pregame analysis: (1) a traditional, scenario-based analysis, common in force planning efforts, and (2) a novel analysis using RAND's Robust Decision Making (RDM) method. The type of analysis presented appeared to influence the decisionmaking process and resulting decisions; however, the influence of player experience seemed to be larger than that of the analysis presented. The researchers found that the analytic community could use such structured comparison games to assess how policymakers ingest and use analysis. This kind of approach could increase the utility of policy recommendations in the future by providing a window into how analysis is interpreted and used by decisionmakers.
- Type of analysis appeared to affect the decisionmaking process. In the game prompted by scenario-based analysis, players spent considerable time debating the validity of underlying assumptions, arguing that the assumed demands for, and supply of, forces were optimistic and that shortfalls were larger than presented. In contrast, in the game informed by RDM analysis, players debated the likelihood of different scenarios, the importance of the different scenarios, and the implied risk of the options.
- The type of analysis — scenario-based versus RDM &mdash also had some effect on what decision was eventually made, but the effect was not strong.
- Player experience seemed to have a larger effect on the decisionmaking process and resulting decisions than did the type of analysis presented to the players. In the experimental and control conditions, both the content of the debate and the decisions differed between groups of mid-level and senior officials.
- Games of this design could be used to help research teams hone the communication of analytical approaches and important findings to decisionmakers.
- The game described here focuses solely on the final decisionmaking process and resulting decisions. In principle, future work could examine the effect of a new approach to decision analysis on elements of force planning, such as scenario selection, staffing during baseline assessments, and staffing in the lead-up to a major decision meeting.
- More repetitions of the game will be required to better understand the value proposition of analysis for decisionmaking. In addition, access to a wider population of players who better represent the population of decisionmakers would improve the credibility of game results.
- The analytic community could use such structured comparison games to assess how policymakers ingest and use analysis. This kind of approach could increase the utility of policy recommendations in the future by providing a window into how analysis is interpreted and used by decisionmakers.
Table of Contents
A Game Design for Examining the Effect of Information on Decisionmaking and the Game's Application to a Force Structure Problem
Analysis and Observations
Conclusions and Future Directions to Improve the Structured Comparison Design and Our Prototype Force Planning Game
Description of a Deputy's Management Action Group
Guidance on the Positions of Players in the Deputy's Management Action Group
Move 2 Scenarios
Detailed Thematic Analysis of Move 1 Discussion