As someone who struggles to set up my home Wi-Fi network, I had a surprise invitation to speak on a panel on hypothesis testing for escalation and deterrence in cyberspace. After assuring me I would be weighing in on gaming, not cyber, the panel chair posed the following question: “How might you use wargaming to test hypotheses on cyber, deterrence and escalation control?” Because using games in hypothesis testing applies to domains other than cyber, my answers here have been adapted to be more general.
What is “gaming”?
Deputy Secretary of Defense Bob Work's February 2015 memo on reinvigorating wargaming has spurred a great deal of interest among defense circles. But what the U.S. defense community means by “wargaming” is a very broad set of activities that cover everything from group discussions, planning exercises, training exercises, and meetings that identify requirements and gaps.
I distinguish wargames by this key question: Are there are at least two opposing sides, with an equal opportunity to affect outcomes through their decisions? The more an event is about reviewing the internal processes of one side, rather than examining the consequences that player decisions are having on the course of events, the more it is about planning than gaming. There is nothing wrong with planning. It is simply a different tool.
Another important question is: What proportion of time do participants spend engaging in first-person role play versus third-person commentary about their topics of expertise? Signs of role-playing include immersion in the role's perspective, first-person dialogue, emotional engagement, and active attempts to further the role's objectives. My favorite example of this was watching a State Department participant play a host nation government in a game. With a flushed face, raised voice, and adamant hand gestures, she said, “They [the U.S.] just can't do this. This is our country!” She was so engaged in role playing that she was angry on behalf of a fictitious government—against the government she was part of in real life. The less role-playing within the context of a specific scenario—and the more participant commentary that takes place outside of a role—the less it is a game and the more it is an expert panel discussion. Again, there is nothing wrong with expert panel discussion—it is again simply a different tool.
How does gaming help test your theory?
The act of designing a game will force you to articulate your theory or to be more specific about it. It will also require you to operationalize your variables and theoretical constructs of interest into a specific context, and prompt you to anticipate the ways in which it may play out in that scenario. For example, if your theory says that countries balance against perceived threats, whose perceptions within a country are important? What exactly is significant enough to constitute a threat? What is the time scale involved for threats to be perceived and for reactions to occur? The choice of roles, the levels of analysis, the design of the adjudication mechanism, how long players will have for how many turns, how many turns your game has to cover—all have the practical effect of dragging your theory from the abstract to the tangible. This exercise can also assist you in thinking through potential real-world indicators that are relevant for your theory.
It is important to note that a wargame cannot prove your theory or concept. There are too many variables involved to draw firm conclusions about causal mechanisms and too many questions about the external validity of wargames to be confident that what happens in a game applies to reality. It is impossible to include the full range of important real world factors for the wargame to approach reality, and real life tends to unfold in ways deemed unlikely by the experts.
A game is a model, and all models are abstractions from reality. Having a theory or concept show success in any one model or simulation, such as a wargame, is by itself insufficient proof that it will be successful in the real world. Wargames suffer the additional uncertainty that the same game played even by the same players may produce a different result on another occasion, and in ways not easily understood. This distinguishes it from most forms of computational or mathematical modeling and simulation, where results can differ over repeated runs of the same set of parameters, but in fundamentally knowable ways.
Additionally, there can be considerable conscious and subconscious pressure to give wargame sponsors the outcomes that they want or largely expect. Participants hostile to the concepts ostensibly under scrutiny are not usually invited to wargames sponsored by the concept developer. There can also be considerable pressure on sponsors to declare their game a success. Because of this, caution is warranted if you are both the concept or theory developer, and the wargame sponsor. You would need to combine the outcomes of many games, studies, and other methods to argue that your theory is plausible.
Games can also help by disproving your theory. If you can plausibly construct a good scenario and a believable sequence of events that provide results counter to what your theory predicts, your theory may be wrong. Whether a theory is actually disproved depends a great deal on the game mechanics and the quality of play in the game.
A game can also help test your theory by bringing attention to things that previously did not come to mind. Thomas Schelling, an early developer of nuclear crisis games for the Kennedy administration and Nobel Prize winner in economics for his work on cooperative game theory, remarked that no person, “no matter how rigorous his analysis or heroic his imagination” can “draw up a list of things that would never occur to him.”[1] A good game will confront you with alternative variables, mechanisms and outcomes that may have never occurred to you and which your theory probably doesn't cover.
Wargames also provide a chance to consider alternate conceptualizations of the problem. According to educational psychology, one of the costs of expertise can be cognitive rigidity.[2] Experts are valuable because they have developed deep knowledge in their chosen areas, but sometimes rely heavily on mental models not appropriate for areas outside their expertise. In areas that are poorly understood, such as cyber, this automatic “copy and paste” of mental models is likely to happen more often, but less likely to be directly challenged by others. None of us are exempt from the tendency to give precedence to techniques and methods that have earned us success in the past, or are free from cognitive biases and disciplinary paradigms. Wargaming can give us the opportunity to examine our mental models in entirely new ways.
Gaming can also help test your theory by confronting you with the level of context that you need to take into consideration. Many of us social scientists were taught from our youth to revere parsimonious causal theories as the pinnacle of elegance and intellectual achievement. However, the older I get, the more I am convinced that context greatly matters—and that parsimonious theory is often unhelpful in real-world decision-making. In life, it always depends. A game at least attempts to consider some of this real-life messiness that is context, often in ways that have not occurred to you. Understanding how your theory or concept interacts in specific contexts is immensely important if your work is supposed to have practical implications for a particular organization. Context appears very important in cyber, and the legitimate outcome of many a game may be that context drives more of the outcome than parsimonious theory or quantitative predictions.
Potential pitfalls
In the nine years I have been involved in wargaming in various roles, I have made all of the following mistakes. I encourage anyone interested to learn from them.
Trying to do too much with one game. Keep your list of game objectives as short as possible. I highly recommend that you have ONE objective if you can. Multiple objectives often seriously compromise game design and game play, potentially resulting in nothing being done particularly well or receiving adequate attention. Unfortunately, this rarely happens for a host of reasons. It's easy to get excited about sponsoring a game and tempting to ask for as many things as possible. It's genuinely hard to know ahead of time what you should exclude from a game, because if you knew what you don't need to consider, you probably don't need a game to begin with.
If the wargame has a chance at affecting anything down the line regarding important stakeholders, those stakeholders may require certain types of outputs from the wargame, adding to your objectives. Others in your organization may ask that the game try to solve different problems. So try not to frustrate your wargame by asking for too many things. Narrow the scope and keep repeating to interested parties that their concern is absolutely important, but beyond the scope of this wargame.
Spanning different levels of a phenomenon at once.[3] This is very tricky to do well. National security games often try to straddle the operational and strategic levels at the same time. This is because sponsoring organizations often must answer operational level questions, but are operating in a strategic context that is not static. Trying to generate game play that incorporates dynamic developments at both levels poses challenges for designers as well as players.[4] It is not impossible to have games straddle different levels, but can be tricky for the different levels to receive equal consideration in the game.
Counting on certain outputs from a game. A game can be the wrong tool to generate certain types of products. It is best to rely on a combination of studies, games, and other inputs for concrete recommendations on a specific decision, rather than the outputs of a single game or even series of games. Games not do not produce comprehensive lists of requirements or shortfalls for an organization, given the path-dependent nature of the overall results. Other approaches, such as structured brainstorming with key stakeholders, would produce a better answer in this case than simulating the enemy actions to one course of action. Additionally, if your requirement is to “validate” something, a wargame is likely the wrong tool. When used well, wargames can generate a list of things that you hadn't considered. This list may help identify important questions and factors previously overlooked in studies, analyses, brainstorming, computer models, and live exercises. But wargames are not a substitute for any of these.
Reading too much into a single game. Because a good game can be so engaging, people can put too much stock in what they took away from a single game, particularly if they do not often participate in games. Games can be powerful because they are immersive and “experiential.” Games are also effective forms of engagement because gaming taps into the underlying structures in our brains that we used to learn about and explore a huge number of complex topics while we were younger.
As children, we learn through play. Play—imaginative play—is considered a key component of early childhood learning. A number of therapies aimed at children also incorporate play to help with skill development and emotional processing. The positive relationship between play and cognitive development has also been studied in animals. My personal, completely unscientific theory is that when we wargame, we still tap into these deep and powerful structures in the brain that help us process, imagine, and create. Children learn faster than adults and are more creative. And if play is the primary cognitive tool we learned while younger to process vast amounts of information about the world around us, we could do significantly worse than to use this tool again.
That said, if games are powerful because they tap into our deep and powerful structures in the brain for learning, then games that give “wrong” lessons may be a powerful and persuasive experience for attendees. To mitigate this, it may be interesting to ask during the “hotwash” post-game discussion: What would have to be the case for the group's conclusions about the wargame to be wrong?
Gaming advice
If you are interested in gaming, you already know more about it than you likely give yourself credit for. Model United Nations, board games, card games, miniatures games—if you've taken part in any of these, you have experience that will translate to gaming serious issues.
To quote Peter Perla, the man who literally wrote the book on wargaming, players make the game. One thing that is very true in professional games—particularly for difficult problems with high degrees of uncertainty—is that the knowledge and expertise of the players is paramount.[5] This is definitely true when the purpose of a game is to explore an issue or test a theory. (Games meant to teach or train are obviously geared to those with less knowledge.) Many professional games do poorly not because of design, but because those who come to the game are insufficiently versed in some critical aspect of the problem to properly play out the dynamic. Doing research to find good players is critical, because the players frame the problem. To find players, start early, make yourself known in circles you want to recruit from, show up at conferences, ask experts to refer other experts, and actively woo the highest quality of experts you can.
Also, the more pressure there is on a game to produce a certain set of products, the less you actually want a game and the more you may want to consider another type of event, such as structured discussions or other form of expert data collection. In addition, remember the many types of things that games will not give you. With this in mind, use games to triangulate with multiple other approaches to examine your problem from different angles.
Keep your game under the radar and out of the limelight as much as possible. It may seem a positive to have a high-profile game that gets lots of attention, but high visibility can mean greater scrutiny and pushback even before your game begins. There can also be greater pressure for more predictable sets of products and greater certainty in what the game will produce.
And remember to look for the markers of successful games. Active engagement and the right roster of players are not enough to result in a good game. In a truly successful game, what you should see are leading experts in their field immersed in role playing and thinking of things that had not occurred to them before.
Notes
- [1] Thomas Schelling, “The role of war games and exercises,” in A. Carter, J. Steinbruner, & Zraket, C. (Eds.), Managing Nuclear Operations (Washington, DC: The Brookings Institution, 1987), pp. 426-444.
- [2]Gregory Schraw, “Knowledge: Structures and Processes,” in Handbook of Educational Psychology, 2nd edition (American Psychological Association, 2006), edited by Patricia A. Alexander and Philip H. Winne, p. 259.
- [3] Elizabeth Bartels, Margaret McCown, and Timothy Wilkie, “Designing Peace and Conflict Exercises: Levels of Analysis, Scenario, and Role Specification,” Simulation and Gaming (February 2013), Vol. 44, No. 1, pp. 36-50.
- [4] One common result is that players will choose one of the levels, as noted by Ellie Bartels at the Pardee RAND Graduate School. I have also seen one team in a game pick the strategic level in a game, and the other team pick the operational level, with both declaring victory in the end.
- [5] Some people also make better players than others—a topic where I do not believe we have a lot of research.
Yuna Huh Wong is a researcher for the RAND Corporation where she is involved with wargames, futures methods, and a variety of defense-related studies. She also sits on the Board of Directors for the Military Operations Research Society; and is a Research/Affiliate Scholar at the Center for the Study of Narrative and Conflict Resolution at the School for Conflict Analysis and Resolution at George Mason University.
This article was adapted from comments she made during a Dec. 3, 2015, panel on “Testing Hypotheses: Escalation and Deterrence in Cyberspace,” at the Cyberspace and Deterrence Academic and Inter-Agency Symposium at the Elliot School of International Affairs, George Washington University, Washington, D.C
This commentary originally appeared on PAXsims on May 18, 2016. Commentary gives RAND researchers a platform to convey insights based on their professional expertise and often on their peer-reviewed research and analysis.
