RDM Step: Decision Framing

Although RDM is iterative, most analyses begin with decision framing, a deliberation step in which stakeholders develop a shared understanding of the problem they are trying to solve, the information they need to do so, and the analysis that will get them that information.

The four key elements of an RDM analysis are summarized in a two-by-two matrix, called an XLRM matrix and introduced by Lempert, Popper, and Bankes, 2003 (see Table 1).

As the steps of the RDM analysis progress, the stakeholders might refine the decision framing, for example by developing new strategies, evaluating the implications of new uncertainties or focusing on certain portions of the uncertainty space, and adding or refining metrics.

Consider a long-term planning effort by a small water utility whose raw water supply comes from a river intake that fills its supply reservoir (see Table 2). Through a participatory scoping exercise, the utility concluded that it wants to achieve a safe yield, which it defines as raw water storage above 75 percent of capacity for 90 percent or more of the time. The utility has a water systems model to simulate daily demand and supply and track water storage in its system. The utility is concerned about how climate change might affect its system, how future demand might evolve, and what impact potentially new wholesale customers might have. Specifically, the utility has been provided ten climate change projections by its university partner that all suggest warming, but with a range of precipitation changes to wetter and drier conditions. The utility also has developed projections of future demand from a growing population, ranging from a low of 75 million to a high of 120 million gallons per day (mgd). The possibility of new wholesale customers is characterized by the size of the potential contract as large, medium, or small. There is disagreement among stakeholders about the likelihood of the future contract sizes.

RDM Step: Evaluate Strategies Across Futures

The next step of RDM is an analytical one in which the decision framework is executed, i.e., the models (R) are run over the strategies (L) and uncertainties (X) to output the performance of each strategy according to the metrics (M). The uncertainties are sampled to create a set of futures, where one future reflects a specific assumption for each uncertainty. There might be hundreds or thousands of futures necessary to capture combinations of uncertainties across a large uncertainty space. Then, each strategy is evaluated in each of those futures. The result is typically a database of cases, where each case is the performance of a single strategy in a single future. The database often contains thousands of runs, depending on the number of strategies and the number of futures.

There is no single correct number of futures to develop or number of strategies to evaluate. In general, the greater the number of independent uncertainties, the more futures a study should evaluate to sufficiently explore the uncertainty space. With modern computers, computing clusters, and the cloud, computing time might not necessarily be a limiting factor, and it is common to evaluate hundreds to thousands of futures in an RDM study. In some cases, an RDM analysis might use previously developed projections of various parameters. As an example, the Colorado River Basin Case Study evaluated thousands of previously developed sequences of future hydrology derived from historical data, paleo records, and future global climate model projections. In other cases, a sampling strategy would be developed to explore as efficiently as possible across many uncertain parameters. It is common to use a Latin hypercube sampling approach when uncertainties are single factors, as opposed to transient time series. Several of the case studies discussed in this tool illustrate some of the benefits and costs of evaluating only a small set of futures.

In our example, the utility developed an experimental design to evaluate a wide range of futures using its systems model. It developed 20 futures that uniformly sample across the ten climate projections, projections of future customer demand between 75 mgd and 120 mgd, and three sizes of possible wholesale contracts (small, medium, or large). (For simplicity, we have intentionally left the meaning of these contract sizes undefined.) The utility evaluated the performance of its existing system across these 20 futures and calculated the key performance metric (safe yield) for each future. The utility also had been considering an adaptation strategy focused on efficiency. It evaluated that strategy under the same futures. The cost of the strategy was the same across all futures because it relied on the same set of actions, regardless of future conditions. This information was compiled into a spreadsheet that is summarized in Visualization 1. As an example, Future 1 included a projection of total demand of 103 mgd, climate projection 7 (warmer and drier), and moderate expansion of the wholesale customer base. For that future, the safe yield for the baseline strategy was calculated to be 78 percent.

RDM Step: Vulnerability Analysis

A vulnerability analysis asks, “Under which sets of uncertain conditions does each strategy fail to meet stakeholder goals?” In our water utility example, a strategy fails in any future in which the safe yield is below 90 percent. Although it is tempting to count the number of futures in which each strategy is successful, the database is meant to explore uncertainties and not predict which ones are likely. Therefore, probabilistic interpretations of the database should generally be avoided. Instead, analysts ask, “In what kinds of futures does a particular strategy fail?”

Because of its thousands of cases involving dozens of strategies, uncertainties, and metrics, this kind of eyeball analysis is often impossible. Instead, advanced data-mining techniques can help. Such methods as the Patient Rule Induction Method (PRIM) help find simple explanations for the conditions in which policies fail or succeed. The basic approach of PRIM and similar methods is to distill the large amount of data generated in the evaluate strategies across futures step of the RDM framework down to the most-relevant information for decisionmakers and stakeholders. This requires not only identifying conditions that lead to vulnerabilities but also ensuring that they are easily interpretable. Usually, the smaller the number of conditions used to describe a strategy’s vulnerability, the higher the interpretability is. A more thorough discussion of PRIM and scenario discovery is available in the Frequently Asked Questions section.

Figure 3 shows the result of each future with respect to each of the uncertain factors; the climate projections are on the horizontal axis, water demand estimates are on the vertical axis, and the size of the symbol indicates the wholesale type. The value under each mark is the safe yield result for that combination of factors.

Using PRIM, the utility then identified the conditions in which its baseline strategy is vulnerable. In this case, the utility defined them as

• demand ≥ 95 mgd
• climate projection: 6–10 (warmer and drier projections)
• wholesale type: medium or large.

This vulnerability is identified by a box in Figure 3.

An objective of RDM is to identify strategies that perform better across the uncertainties. So, the utility further identified the conditions in which the proposed adaptation strategy is vulnerable as

• demand ≥ 100 mgd
• climate projection: 8–10 (warmer and drier projections).

Note that only two conditions—demand at 100 mgd or higher and climate projections 8–10—are needed to summarize the vulnerability of the adaptation strategy (see Figure 4).

RDM Step: Trade-Off Analysis

The next step in RDM is deliberative. No strategy is likely to be optimal across all performance measures. In the above example, the adaptation strategy is more robust than the baseline strategy, allowing the utility to meet its goals under higher demands and worse climate conditions than the baseline. It is also more than 150 percent more costly. In some cases, stakeholders might be willing to trade off cost for robustness or vice versa.

If stakeholders find a strategy that makes an acceptable trade-off between different performance measures across a wide enough range of futures, then they might choose that option as a sufficiently robust strategy and end the RDM exercise. This notion of a sufficiently robust strategy depends on stakeholders’ subjective views about what kinds of futures are plausible and how much they must be protected against. However, a conversation about what the future could hold and what range of possibilities can and should be planned for is fundamentally different from and potentially more productive and transparent than one focused on trying to predict the future.

RDM Step: New Futures and Strategies

Often a single iteration of RDM might not yield a sufficiently robust solution, or it might reveal new opportunities for policies or information. For example, the initial case-generation and vulnerability analysis in the evaluate strategies across futures and vulnerability analysis steps might reveal futures of particular interest to the choice of strategy, which are explored in greater detail through new futures. These futures can be analyzed and the process can continue until the parties to the decision reach an acceptable solution.

Guidance on How Best to Apply DMDU

The commentary in this section is drawn from and adapts content in Lempert, Popper, et al., 2013.

Simplified approaches, such as scenario planning, are often a useful step toward a full DMDU analysis. Because analysts, stakeholders, and decisionmakers often are familiar with scenario-based efforts, such efforts can be useful in building and engaging participants in an analytical process that embraces uncertainty. In particular, these approaches help analysts and stakeholders explore answers to the question, “What might happen in our basin?” Examining a handful of diverse cases is a useful way to challenge prior assumptions, especially if the default approach has been to make single predictions of the future.

However, simplified approaches have important limitations. They have limited utility in answering the question, “What should we do in our basin to achieve our goals?” This is because decisionmakers do not know which of the handful of scenarios will come to pass; indeed, none might come to pass, because the scenarios often are chosen to defy predictions and illustrate surprise. And, as the trade-off analysis and new futures and strategies steps of the RDM framework illustrate, the uncertain conditions encapsulated by the scenarios are often too sparse to provide bottom-line conclusions about the strengths and weaknesses of strategies that would be necessary to make tough decisions about which strategy to choose in the near term.

This is where stronger DMDU techniques can help. They use the same models and analytical tools as simplified approaches, but they use them differently. In a simpler approach, much emphasis is placed on crafting the scenarios or storylines, because there can be only a few. This can be seen in the intensive process in which the Pecos River–New Mexico Basin Study team engaged to choose their diverse storylines from among 930 potential futures. In the end, however, stakeholders must select strategies, not scenarios.

In contrast, DMDU approaches move the analytical effort away from crafting scenarios and toward understanding strategies. This is done by running models over all of the futures—i.e., the 930 plausible futures from which the five storylines were chosen—and then using advanced computational tools to analyze the results. The analysis of the database of cases once again reveals a handful of scenarios. Critically, these scenarios are different: They describe the decision-relevant strengths and weaknesses of the strategies, not characteristics of the future. Therefore, these scenarios offer the kind of analytical insights that stakeholders need to make trade-offs and, ultimately, make choices around their options.

The transition to full DMDU analyses does raise potential data, computational, and other technical challenges, which can be overcome in many cases. One potential implementation barrier is that DMDU analyses require more computer processor time than a traditional approach because DMDU analyses conduct hundreds to thousands of runs; they also require more computer storage to save the results. In practice, these are not significant constraints given the widespread availability of cloud computing resources, but using these resources requires some training and incurs some cost.

Configuring a model to run over hundreds to thousands of cases often represents the greater challenge. For instance, complex models might have an input file structure or a graphical user interface that makes automation difficult. Analysts might need to be retrained and rework the model to enable batch runs. Fortunately, this software and training proves to be a sound investment because it is generally useful for a variety of analyses.

Finally, perhaps DMDU’s most-significant implementation challenges arise because it represents a new way of thinking about how near-term actions can best manage future risks. Analysts are generally trained in predictive thinking, and the decisionmakers they inform often expect predictive quantitative information. Simpler methods move from traditional questions about what will happen to questions about what might happen. This is an important step. DMDU goes further and enables analysts and decisionmakers to be decision-oriented by asking, “What should we do today to most effectively manage the full range of events that might happen?” Using DMDU requires training for analysts and a path by which organizations become comfortable using new and more-effective types of quantitative information.

Frequently Asked Questions

How does sampling in RDM differ from probabilistic or scenario-based approaches?

Traditional scenario planning and probabilistic planning use samples to describe what the future will bring. RDM uses samples in a fundamentally different way. The samples are used iteratively to stress-test strategies across a wide range of possible future conditions without making judgments about whether one future is more likely than any other. Thus, analysts sample uniformly across the range of plausible values to ensure that all viewpoints about the future are represented but are not judging whether one sample is more likely than another.

Figure 5 shows this difference. The top pane shows an assumed likelihood distribution for a hypothetical uncertain factor (blue curve). A Monte Carlo probabilistic sampling approach would sample proportionally to a specified likelihood distribution. For the illustrative case shown in the figure, more samples (11) are taken from the left half of the distribution, reflecting the higher assessed likelihoods for that portion of the uncertain factor range. In contrast, only five samples are taken from the right half of the factor range. Because the derived weights will influence the results of the analysis, there must be agreement on the shape of the distribution. In many cases, discussions about the nature of these distributions can take a significant amount of time and even jeopardize the acceptance of the analysis by diverse groups.

The bottom panel shows the sampling across the same factor range for a traditional scenario approach (green, short-dashed lines) and an RDM sampling approach (purple, longer-dashed lines). Note that the traditional scenario approach generally uses a small number of samples, which might or might not be uniformly distributed across the range. Similar to the probabilistic approach, a traditional scenario approach requires important decisions to be made about what values to specify for the small number of scenarios. This also can be contentious because individual stakeholders might advocate for the analysis of those scenarios that they believe will justify their preferred strategy.

SOURCE: Groves et al., 2014.

How do we know when PRIM or other techniques have found the right conditions to describe a vulnerability?

In our earlier water utility example, the conditions that define the baseline vulnerability did not perfectly capture when the baseline strategy met or failed to meet decisionmakers’ goals. In particular, there are two futures in which the vulnerable conditions did not hold and yet the baseline strategy did not meet stakeholder goals. (These are the two red X’s outside the red box in Figure 3).

To capture these futures, a more expansive set of conditions could be used. One might identify the conditions in which the baseline strategy is vulnerable as

• demand ≥ 80 mgd
• climate projection: 4–10 (warmer and drier projections)
• wholesale type: medium or large.

This is shown in Figure 6.

However, these sets of conditions also apply to futures in which the baseline strategy does meet stakeholder goals. (These are the green circles inside the box in Figure 6.) There are no “correct” definitions of vulnerable conditions, and there is a natural tension between how broad or how narrow the definitions should be. How useful a particular definition is can be measured with the following three characteristics:

• coverage: What percentage of futures in which a strategy fails are described by the vulnerable scenario? Ideally, the vulnerability would contain all such cases in the database and coverage would be 100 percent.
• density: What percentage of futures in the vulnerable scenario are ones in which the strategy fails? Ideally, all of the cases within the vulnerability would be failures and density would be 100 percent.
• interpretability: How easy is it for users to understand the vulnerable scenarios? The number of uncertainties used to define the scenario serves as a proxy for interpretability. In general, the smaller the number of uncertainties used, the higher the interpretability is.

Ideally, all of the futures captured in a set of vulnerable conditions would not meet agency goals (i.e., 100-percent density), and the vulnerable conditions would describe all of the futures that do not meet agency goals (i.e., 100-percent coverage). However, this is rarely the case. The first set of conditions in the above example had higher density and lower coverage, while the second set of conditions had lower density but higher coverage.

One additional approach to negotiate between coverage and density is to define multiple sets of conditions. For example, in addition to the original set of conditions, decisionmakers might say that the baseline system is vulnerable if demand is higher than 110 mgd and the climate projection is between 4 and 5. This second set of conditions would capture one of the vulnerable cases that was not captured by the original scenario. Together, these two sets of conditions would have higher density and higher coverage, but they would be less interpretable: There are now several conditions for stakeholders to track, which makes it difficult to intuit how the baseline performs.

How do PRIM and other techniques find vulnerable scenarios?

When there are more than two uncertainty variables, it is difficult to identify vulnerable scenarios through inspection. PRIM can be used to define vulnerable scenarios. PRIM was first used in a DMDU study to identify the vulnerabilities of long-term policies. This specific approach was later more formally described and compared with other methods and continues to be widely used because of ease of use and freely available software in R and Python and because it defines easily interpretable scenarios. PRIM and other analytic scenario-discovery tools are used to (1) identify which uncertainties or characterizations of uncertainty are most important in determining future conditions to which a system is vulnerable and (2) define a concise set of rules that describe the range of uncertainty. These tools are most valuable when there is a large number of evaluations of the future to analyze.

Specifically, PRIM iteratively identifies “boxes,” which are defined by the included range of different input variables that balance coverage, density, and interpretability. The Colorado River Basin Case Study provides an interactive tool that illustrates how PRIM works.

There are other algorithms that can be used to support scenario discovery. Dimensional stacking, for example, is described in Chapter 7 of Marchau et al., 2019, and the C5.0 algorithm is used in the Monterrey, Mexico, Case Study to define adaptive strategies based on the identified vulnerabilities.