Robust Decision Making

Robust Decision Making

Robust Decision Making (RDM) is a widely used approach for Decisionmaking Under Deep Uncertainty (DMDU). Originally developed at the RAND Corporation, RDM asks “How can we make good decisions without first needing to make predictions?” See RAND’s Center for Decision Making Under Uncertainty for additional RDM information and a curated set of studies.

RDM is perhaps best explained by contrast. Traditional planning approaches begin with the question “What will future conditions be?” and then identify a strategy or plan that will perform best for that prediction (see Figure 1). Analysts focus on making accurate predictions and optimizing to that future. They might use sensitivity analyses to identify weaknesses in their strategy. These approaches work well when conditions can be easily predicted and there is little disagreement among stakeholders.

Figure 1. Steps in a Typical Planning Analysis

The real world is almost never like this. For most real policy problems, the future is deeply uncertain and stakeholders do not agree on what it will look like, how policy options will perform in it, or how to value different aspects of performance. When applied to real-world problems, traditional approaches lead to gridlock (when stakeholders cannot agree on the assumptions in an analysis) and brittle solutions (when the future unfolds differently than predicted and the chosen solution fails). Worse still, traditional approaches encourage stakeholders to advocate for assumptions that they know will lead to the strategy or policy that they already prefer. In sum, traditional approaches encourage the kinds of cognitive biases that stymie collaborative, evidence-based decisionmaking.

When should you use such DMDU methods as RDM?

DMDU methods, such as RDM, are appropriate for problems that involve deep uncertainty (Lempert, Popper, and Bankes, 2003, p. 208). These are problems in which the parties to a decision do not know or cannot agree on

  • the system model that relates action to consequences. For example, water resource managers might disagree about whether model A or model B is a better representation of the impact of drought on crop yield and irrigation.
  • the probability distributions to place over the inputs to these models. For example, there is much uncertainty about how climate change will affect precipitation and temperature in any particular area.
  • the value of outcomes. Decisionmakers and stakeholders might disagree about the value of protecting endangered species relative to the value of irrigating crop land, or they might have different preferences between an approach that increases water supply and an approach that decreases demand.

In general, real-world planning problems always involve deep uncertainties because they involve complex and dynamic human and environmental systems for which simple models, reliable predictions, and consensus on values are scarce.

DMDU in general—and RDM specifically—seeks to overcome these failures by focusing stakeholders’ attention on the characteristics of their policy options and not on predictions of the future. RDM asks a different set of questions entirely, such as the following:

  • What are the conditions that would affect how our current or leading strategies perform?
  • Under what conditions does our strategy fail to meet different stakeholders’ goals?
  • Are those conditions sufficiently plausible that we should improve on our strategy?
  • What are the decisions that we have to make now, and which ones can we safely defer for the future?

RDM asks and answers these questions in an iterative process of “deliberation with analysis” (National Research Council, 2009). That is, stakeholder deliberation informs the kinds of analysis that are needed to answer key questions about the policy problem, and the analysis provides information over which stakeholders deliberate. This kind of approach is critical for solving complex and deeply uncertain real-world policy problems.

The following sections describe the steps of an RDM study, which are depicted in Figure 2, and use a running water resource planning vignette to illustrate those steps.

Figure 2. Iterative Steps of Robust Decision Making

  • Decision framing
  • Evaluate strategy across futures
  • Vulnerability analysis
  • Trade-off analysis
  • New futures and strategies

The vulnerability analysis step may produce decision-relevant scenarios. The trade-off analysis step may be skipped or may produce robust strategies. Either trade-off analysis or new futures and strategies can lead back to the first step, decision framing.

Decision framing: Develop the scope of the analysis through structured engagements with stakeholders and decisionmakers. The scope is often summarized using an XLRM matrix, which lists the measures (M) of performance used to evaluate decisions, the levers (L) or actions that can be taken by policymakers, the key uncertain factors (X) that might affect the ability of the decisions to meet objectives, and the mathematical relationships (R) among these factors, usually combined in one or more models.
Evaluate strategies across futures: Different sets of decisions represent strategies, which are then evaluated across many plausible futures, where each future reflects one set of assumptions across the uncertainties.
Vulnerability analysis: Statistical tools are used to identify the future uncertain conditions that would lead the strategies to perform poorly
Decision-relevant scenarios: Concise descriptions of the futures in which strategies perform poorly. These scenarios are decision-relevant; they would necessitate a different strategy to achieve acceptable performance.
Trade-off analysis: Interactive visualizations are used to highlight performance and cost trade-offs among different strategies. This supports the selection of a final robust strategy or provides information for defining additional futures or strategies.
New futures and strategies: Using the vulnerability and trade-off analysis, new futures are defined to either expand the range of uncertainties or focus more on specific combinations of uncertainties that are relevant to the choice of strategy. Strategies can also be refined to be more robust using lessons from the vulnerability analysis.
Robust strategies: A robust strategy is one that performs sufficiently well over a wide range of plausible futures. Iterations through all of the steps in the RDM framework can help develop more-robust strategies.

RDM Step: Decision Framing

Although RDM is iterative, most analyses begin with decision framing, a deliberation step in which stakeholders develop a shared understanding of the problem they are trying to solve, the information they need to do so, and the analysis that will get them that information.

The four key elements of an RDM analysis are summarized in a two-by-two matrix, called an XLRM matrix and introduced by Lempert, Popper, and Bankes, 2003 (see Table 1).

Table 1. XLRM Matrix and Definitions for the Decision Framing RDM Step

(X) Uncertainties (L) Policy Levers or Strategies
  • Uncertain factors that might affect the achievement of desired goals
  • Combinations of assumptions about uncertainties define a plausible future
  • Actions or decisions that can be taken by policymakers
  • Combinations of different levers define a strategy intended to help achieve desired goals
(R) Models or Relationships (M) Performance Metrics or Measures
  • How policy levers and uncertainties interact to affect future outcomes
  • Often represented by a mathematical model
  • Quantitative or qualitative metrics used to evaluate outcomes and the performance of strategies relative to goals

As the steps of the RDM analysis progress, the stakeholders might refine the decision framing, for example by developing new strategies, evaluating the implications of new uncertainties or focusing on certain portions of the uncertainty space, and adding or refining metrics.

Consider a long-term planning effort by a small water utility whose raw water supply comes from a river intake that fills its supply reservoir (see Table 2). Through a participatory scoping exercise, the utility concluded that it wants to achieve a safe yield, which it defines as raw water storage above 75 percent of capacity for 90 percent or more of the time. The utility has a water systems model to simulate daily demand and supply and track water storage in its system. The utility is concerned about how climate change might affect its system, how future demand might evolve, and what impact potentially new wholesale customers might have. Specifically, the utility has been provided ten climate change projections by its university partner that all suggest warming, but with a range of precipitation changes to wetter and drier conditions. The utility also has developed projections of future demand from a growing population, ranging from a low of 75 million to a high of 120 million gallons per day (mgd). The possibility of new wholesale customers is characterized by the size of the potential contract as large, medium, or small. There is disagreement among stakeholders about the likelihood of the future contract sizes.

Table 2. Example of XLRM Matrix for RDM

(X) Uncertainties (L) Policy Levers or Strategies
  • Climate conditions
  • Demand from existing customers
  • Contract size for a new wholesale customer
  • Baseline: current infrastructure
  • Adaptation: invest in efficiency
(R) Models or Relationships (M) Performance Metrics or Measures
  • Small water utilities water system model
  • Safe yield
  • Cost of strategy

RDM Step: Evaluate Strategies Across Futures

The next step of RDM is an analytical one in which the decision framework is executed, i.e., the models (R) are run over the strategies (L) and uncertainties (X) to output the performance of each strategy according to the metrics (M). The uncertainties are sampled to create a set of futures, where one future reflects a specific assumption for each uncertainty. There might be hundreds or thousands of futures necessary to capture combinations of uncertainties across a large uncertainty space. Then, each strategy is evaluated in each of those futures. The result is typically a database of cases, where each case is the performance of a single strategy in a single future. The database often contains thousands of runs, depending on the number of strategies and the number of futures.

There is no single correct number of futures to develop or number of strategies to evaluate. In general, the greater the number of independent uncertainties, the more futures a study should evaluate to sufficiently explore the uncertainty space. With modern computers, computing clusters, and the cloud, computing time might not necessarily be a limiting factor, and it is common to evaluate hundreds to thousands of futures in an RDM study. In some cases, an RDM analysis might use previously developed projections of various parameters. As an example, the Colorado River Basin Case Study evaluated thousands of previously developed sequences of future hydrology derived from historical data, paleo records, and future global climate model projections. In other cases, a sampling strategy would be developed to explore as efficiently as possible across many uncertain parameters. It is common to use a Latin hypercube sampling approach when uncertainties are single factors, as opposed to transient time series. Several of the case studies discussed in this tool illustrate some of the benefits and costs of evaluating only a small set of futures.

In our example, the utility developed an experimental design to evaluate a wide range of futures using its systems model. It developed 20 futures that uniformly sample across the ten climate projections, projections of future customer demand between 75 mgd and 120 mgd, and three sizes of possible wholesale contracts (small, medium, or large). (For simplicity, we have intentionally left the meaning of these contract sizes undefined.) The utility evaluated the performance of its existing system across these 20 futures and calculated the key performance metric (safe yield) for each future. The utility also had been considering an adaptation strategy focused on efficiency. It evaluated that strategy under the same futures. The cost of the strategy was the same across all futures because it relied on the same set of actions, regardless of future conditions. This information was compiled into a spreadsheet that is summarized in Visualization 1. As an example, Future 1 included a projection of total demand of 103 mgd, climate projection 7 (warmer and drier), and moderate expansion of the wholesale customer base. For that future, the safe yield for the baseline strategy was calculated to be 78 percent.

Visualization 1. Example of Experimental Design and Metric Outcomes for RDM

RDM Step: Vulnerability Analysis

A vulnerability analysis asks, “Under which sets of uncertain conditions does each strategy fail to meet stakeholder goals?” In our water utility example, a strategy fails in any future in which the safe yield is below 90 percent. Although it is tempting to count the number of futures in which each strategy is successful, the database is meant to explore uncertainties and not predict which ones are likely. Therefore, probabilistic interpretations of the database should generally be avoided. Instead, analysts ask, “In what kinds of futures does a particular strategy fail?”

Because of its thousands of cases involving dozens of strategies, uncertainties, and metrics, this kind of eyeball analysis is often impossible. Instead, advanced data-mining techniques can help. Such methods as the Patient Rule Induction Method (PRIM) help find simple explanations for the conditions in which policies fail or succeed. The basic approach of PRIM and similar methods is to distill the large amount of data generated in the evaluate strategies across futures step of the RDM framework down to the most-relevant information for decisionmakers and stakeholders. This requires not only identifying conditions that lead to vulnerabilities but also ensuring that they are easily interpretable. Usually, the smaller the number of conditions used to describe a strategy’s vulnerability, the higher the interpretability is. A more thorough discussion of PRIM and scenario discovery is available in the Frequently Asked Questions section.

Figure 3 shows the result of each future with respect to each of the uncertain factors; the climate projections are on the horizontal axis, water demand estimates are on the vertical axis, and the size of the symbol indicates the wholesale type. The value under each mark is the safe yield result for that combination of factors.

Figure 3. Example Outcomes for Baseline Strategy and Defined Vulnerability for RDM

Using PRIM, the utility then identified the conditions in which its baseline strategy is vulnerable. In this case, the utility defined them as

  • demand ≥ 95 mgd
  • climate projection: 6–10 (warmer and drier projections)
  • wholesale type: medium or large.

This vulnerability is identified by a box in Figure 3.

An objective of RDM is to identify strategies that perform better across the uncertainties. So, the utility further identified the conditions in which the proposed adaptation strategy is vulnerable as

  • demand ≥ 100 mgd
  • climate projection: 8–10 (warmer and drier projections).

Note that only two conditions—demand at 100 mgd or higher and climate projections 8–10—are needed to summarize the vulnerability of the adaptation strategy (see Figure 4).

Figure 4. Example Outcomes for Adaptation Strategy and Defined Vulnerability for RDM

RDM Step: Trade-Off Analysis

The next step in RDM is deliberative. No strategy is likely to be optimal across all performance measures. In the above example, the adaptation strategy is more robust than the baseline strategy, allowing the utility to meet its goals under higher demands and worse climate conditions than the baseline. It is also more than 150 percent more costly. In some cases, stakeholders might be willing to trade off cost for robustness or vice versa.

If stakeholders find a strategy that makes an acceptable trade-off between different performance measures across a wide enough range of futures, then they might choose that option as a sufficiently robust strategy and end the RDM exercise. This notion of a sufficiently robust strategy depends on stakeholders’ subjective views about what kinds of futures are plausible and how much they must be protected against. However, a conversation about what the future could hold and what range of possibilities can and should be planned for is fundamentally different from and potentially more productive and transparent than one focused on trying to predict the future.

RDM Step: New Futures and Strategies

Often a single iteration of RDM might not yield a sufficiently robust solution, or it might reveal new opportunities for policies or information. For example, the initial case-generation and vulnerability analysis in the evaluate strategies across futures and vulnerability analysis steps might reveal futures of particular interest to the choice of strategy, which are explored in greater detail through new futures. These futures can be analyzed and the process can continue until the parties to the decision reach an acceptable solution.

Guidance on How Best to Apply DMDU

The commentary in this section is drawn from and adapts content in Lempert, Popper, et al., 2013.

Simplified approaches, such as scenario planning, are often a useful step toward a full DMDU analysis. Because analysts, stakeholders, and decisionmakers often are familiar with scenario-based efforts, such efforts can be useful in building and engaging participants in an analytical process that embraces uncertainty. In particular, these approaches help analysts and stakeholders explore answers to the question, “What might happen in our basin?” Examining a handful of diverse cases is a useful way to challenge prior assumptions, especially if the default approach has been to make single predictions of the future.

However, simplified approaches have important limitations. They have limited utility in answering the question, “What should we do in our basin to achieve our goals?” This is because decisionmakers do not know which of the handful of scenarios will come to pass; indeed, none might come to pass, because the scenarios often are chosen to defy predictions and illustrate surprise. And, as the trade-off analysis and new futures and strategies steps of the RDM framework illustrate, the uncertain conditions encapsulated by the scenarios are often too sparse to provide bottom-line conclusions about the strengths and weaknesses of strategies that would be necessary to make tough decisions about which strategy to choose in the near term.

This is where stronger DMDU techniques can help. They use the same models and analytical tools as simplified approaches, but they use them differently. In a simpler approach, much emphasis is placed on crafting the scenarios or storylines, because there can be only a few. This can be seen in the intensive process in which the Pecos River–New Mexico Basin Study team engaged to choose their diverse storylines from among 930 potential futures. In the end, however, stakeholders must select strategies, not scenarios.

In contrast, DMDU approaches move the analytical effort away from crafting scenarios and toward understanding strategies. This is done by running models over all of the futures—i.e., the 930 plausible futures from which the five storylines were chosen—and then using advanced computational tools to analyze the results. The analysis of the database of cases once again reveals a handful of scenarios. Critically, these scenarios are different: They describe the decision-relevant strengths and weaknesses of the strategies, not characteristics of the future. Therefore, these scenarios offer the kind of analytical insights that stakeholders need to make trade-offs and, ultimately, make choices around their options.

The transition to full DMDU analyses does raise potential data, computational, and other technical challenges, which can be overcome in many cases. One potential implementation barrier is that DMDU analyses require more computer processor time than a traditional approach because DMDU analyses conduct hundreds to thousands of runs; they also require more computer storage to save the results. In practice, these are not significant constraints given the widespread availability of cloud computing resources, but using these resources requires some training and incurs some cost.

Configuring a model to run over hundreds to thousands of cases often represents the greater challenge. For instance, complex models might have an input file structure or a graphical user interface that makes automation difficult. Analysts might need to be retrained and rework the model to enable batch runs. Fortunately, this software and training proves to be a sound investment because it is generally useful for a variety of analyses.

Finally, perhaps DMDU’s most-significant implementation challenges arise because it represents a new way of thinking about how near-term actions can best manage future risks. Analysts are generally trained in predictive thinking, and the decisionmakers they inform often expect predictive quantitative information. Simpler methods move from traditional questions about what will happen to questions about what might happen. This is an important step. DMDU goes further and enables analysts and decisionmakers to be decision-oriented by asking, “What should we do today to most effectively manage the full range of events that might happen?” Using DMDU requires training for analysts and a path by which organizations become comfortable using new and more-effective types of quantitative information.

Frequently Asked Questions

How does sampling in RDM differ from probabilistic or scenario-based approaches?

Traditional scenario planning and probabilistic planning use samples to describe what the future will bring. RDM uses samples in a fundamentally different way. The samples are used iteratively to stress-test strategies across a wide range of possible future conditions without making judgments about whether one future is more likely than any other. Thus, analysts sample uniformly across the range of plausible values to ensure that all viewpoints about the future are represented but are not judging whether one sample is more likely than another.

Figure 5 shows this difference. The top pane shows an assumed likelihood distribution for a hypothetical uncertain factor (blue curve). A Monte Carlo probabilistic sampling approach would sample proportionally to a specified likelihood distribution. For the illustrative case shown in the figure, more samples (11) are taken from the left half of the distribution, reflecting the higher assessed likelihoods for that portion of the uncertain factor range. In contrast, only five samples are taken from the right half of the factor range. Because the derived weights will influence the results of the analysis, there must be agreement on the shape of the distribution. In many cases, discussions about the nature of these distributions can take a significant amount of time and even jeopardize the acceptance of the analysis by diverse groups.

The bottom panel shows the sampling across the same factor range for a traditional scenario approach (green, short-dashed lines) and an RDM sampling approach (purple, longer-dashed lines). Note that the traditional scenario approach generally uses a small number of samples, which might or might not be uniformly distributed across the range. Similar to the probabilistic approach, a traditional scenario approach requires important decisions to be made about what values to specify for the small number of scenarios. This also can be contentious because individual stakeholders might advocate for the analysis of those scenarios that they believe will justify their preferred strategy.

Figure 5. Comparison of Probabilistic Sampling and Traditional Scenarios with an RDM Sampling Approach

SOURCE: Groves et al., 2014.

How do we know when PRIM or other techniques have found the right conditions to describe a vulnerability?

In our earlier water utility example, the conditions that define the baseline vulnerability did not perfectly capture when the baseline strategy met or failed to meet decisionmakers’ goals. In particular, there are two futures in which the vulnerable conditions did not hold and yet the baseline strategy did not meet stakeholder goals. (These are the two red X’s outside the red box in Figure 3).

To capture these futures, a more expansive set of conditions could be used. One might identify the conditions in which the baseline strategy is vulnerable as

  • demand ≥ 80 mgd
  • climate projection: 4–10 (warmer and drier projections)
  • wholesale type: medium or large.

This is shown in Figure 6.

Figure 6. Example Outcomes for Baseline Strategy and Alternative Vulnerability for RDM

However, these sets of conditions also apply to futures in which the baseline strategy does meet stakeholder goals. (These are the green circles inside the box in Figure 6.) There are no “correct” definitions of vulnerable conditions, and there is a natural tension between how broad or how narrow the definitions should be. How useful a particular definition is can be measured with the following three characteristics:

  • coverage: What percentage of futures in which a strategy fails are described by the vulnerable scenario? Ideally, the vulnerability would contain all such cases in the database and coverage would be 100 percent.
  • density: What percentage of futures in the vulnerable scenario are ones in which the strategy fails? Ideally, all of the cases within the vulnerability would be failures and density would be 100 percent.
  • interpretability: How easy is it for users to understand the vulnerable scenarios? The number of uncertainties used to define the scenario serves as a proxy for interpretability. In general, the smaller the number of uncertainties used, the higher the interpretability is.

Ideally, all of the futures captured in a set of vulnerable conditions would not meet agency goals (i.e., 100-percent density), and the vulnerable conditions would describe all of the futures that do not meet agency goals (i.e., 100-percent coverage). However, this is rarely the case. The first set of conditions in the above example had higher density and lower coverage, while the second set of conditions had lower density but higher coverage.

One additional approach to negotiate between coverage and density is to define multiple sets of conditions. For example, in addition to the original set of conditions, decisionmakers might say that the baseline system is vulnerable if demand is higher than 110 mgd and the climate projection is between 4 and 5. This second set of conditions would capture one of the vulnerable cases that was not captured by the original scenario. Together, these two sets of conditions would have higher density and higher coverage, but they would be less interpretable: There are now several conditions for stakeholders to track, which makes it difficult to intuit how the baseline performs.

How do PRIM and other techniques find vulnerable scenarios?

When there are more than two uncertainty variables, it is difficult to identify vulnerable scenarios through inspection. PRIM can be used to define vulnerable scenarios. PRIM was first used in a DMDU study to identify the vulnerabilities of long-term policies. This specific approach was later more formally described and compared with other methods and continues to be widely used because of ease of use and freely available software in R and Python and because it defines easily interpretable scenarios. PRIM and other analytic scenario-discovery tools are used to (1) identify which uncertainties or characterizations of uncertainty are most important in determining future conditions to which a system is vulnerable and (2) define a concise set of rules that describe the range of uncertainty. These tools are most valuable when there is a large number of evaluations of the future to analyze.

Specifically, PRIM iteratively identifies “boxes,” which are defined by the included range of different input variables that balance coverage, density, and interpretability. The Colorado River Basin Case Study provides an interactive tool that illustrates how PRIM works.

There are other algorithms that can be used to support scenario discovery. Dimensional stacking, for example, is described in Chapter 7 of Marchau et al., 2019, and the C5.0 algorithm is used in the Monterrey, Mexico, Case Study to define adaptive strategies based on the identified vulnerabilities.