Colorado River Basin Case Study

Case Study Contents

Photo by HKPNC / Getty Images

The Colorado River Basin Water Supply and Demand Study (CRBS), which was published in 2012, was the first of a series of Reclamation-led basin studies examining water management conditions and adaptation options over the next five decades (through 2060). The CRBS was conducted by Reclamation, representatives from the seven U.S. Colorado River Basin states, and a consulting team, in collaboration with a variety of stakeholders (e.g., tribal representatives, conservation organizations, municipal water providers, irrigation districts). Researchers explicitly used Robust Decision Making (RDM) to support their evaluation of the Colorado River Basin’s supply-and-demand imbalance and compare portfolios of management options under a wide variety of plausible future climate, demand, and operations conditions.

The CRBS was the first large-scale water planning project to use the RDM Decisionmaking Under Deep Uncertainty (DMDU) approach to structure an analysis of long-term water supply and demand, and it introduced these new concepts in the midst of a large and complex planning process. After engaging with Reclamation planners, the CRBS team used the RDM process to (1) evaluate a large set of uncertain futures, (2) summarize system performance and identify system vulnerabilities, (3) develop new management options in an interactive way to reflect different stakeholder perspectives, and (4) identify which options would be needed and when across the range of futures evaluated. The RDM analysis also identified several key near-term management options that subsequently were studied in greater depth.

Some participants, however, found that the specific vulnerability analysis techniques were not easily understood and that there was not sufficient time to process the final trade-off analyses. Furthermore, because the methodology was new, it was difficult for participants to explain to others how the analysis was proceeding and, in some cases, how best to interpret the results. This points to the importance of carefully designing an RDM study, taking the necessary time to implement it deliberately, and fully educating the participants regarding the process and its value.

This case study presents the RDM analysis developed for the CRBS and documented in Groves et al., 2013. In addition, it includes some analysis conducted as part of a later workshop organized by RAND Corporation and Lawrence Livermore National Laboratory (LLNL) researchers to explore how high-performance computing could support water resources planning. Many of the original CRBS stakeholders participated in this workshop.

Since the Colorado River Basin Study, Reclamation has continued to explore ways to use DMDU methods to evaluate uncertainty in Colorado River planning. For example, Many-Objective Robust Decision Making (MORDM) is being explored to look at different system operation rules. A follow-on RDM analysis could be used to define additional investment and management strategies to reduce Colorado River vulnerabilities.

Background

The Colorado River is vitally important to the Western United States, providing water and power for 40 million people, including 22 federally recognized tribes, across seven states. It supports billions of dollars annually in economic activity; irrigates 15 percent of U.S. crops (5 million farm acres); and is a lifeline for two dozen national parks, wildlife refuges, and recreation areas.

As the map in Figure 1 shows, the Colorado River Basin includes far more than just the river itself. It is a hydrologic system that spans seven basin states and two states in Northern Mexico. In the Upper Basin are Colorado, Utah, Wyoming, and New Mexico. The Lower Basin includes California, Arizona, and Nevada. The system also includes the main river, tributary streams and rivers, and water storage and delivery infrastructure—dams and reservoirs, hydropower facilities, canals, aqueducts, and pumps. At the heart of this system are two gigantic reservoirs: Lake Powell (which can hold up to 24.3 million acre-feet [MAF] of water) and Lake Mead (which can hold up to 25.9 MAF). Much of the Colorado River Basin’s infrastructure is operated and maintained by Reclamation, which ensures that major water users reliably receive their water deliveries each year while supporting other objectives, including hydropower production, recreation, and environmental flows.

How Colorado River water is allocated to its different uses and users is determined by the 1922 Law of the River, which specifies how much of the then-presumed 16.4–MAF per year supply could be claimed by the seven basin states and Mexico. River conditions in the decades since, however, have revealed that the average available supply over the past century has been about 15 MAF per year—about 9 percent lower than the Law of the River presumption. As a result, the river is overallocated, and this overallocation, combined with the ongoing drought that began in 2000, has depleted storage in Lakes Powell and Mead to almost critical levels. Additional agreements, such as the 2007 Interim Guidelines and the 2019 Drought Contingency Plans, provide for the coordinated operation of Lakes Powell and Mead, incentivize conservation, and specify water delivery reductions under low reservoir conditions.

Map of Colorado River Basin showing Upper and Lower Basin regions. — **Figure 1.** Colorado River Basin Study Area

In January 2010, Reclamation and representatives of the seven states that use Colorado River water began a study of the long-term future of the Colorado River Basin system. The CRBS, which was released in December 2012, was the first collaborative and systematic evaluation of the vulnerabilities of the Colorado River Basin management system, and it developed management strategies to reduce those vulnerabilities. This effort sought to grapple with the many deep uncertainties about future Colorado River conditions, including future surface flows, water demands by the basin states, and even how the system will be operated in the future. The CRBS explicitly used RDM to help address deep uncertainty in its evaluation of the Colorado River Basin’s supply-and-demand imbalance and to develop and compare portfolios of management options (see Figure 2).

Figure 2. Robust Decision Making Steps Used in Colorado River Basin Case Study

Decision Framing
Evaluate Strategies Across Futures
Vulnerability Analysis
Tradeoff Analysis
New Futures and Strategies

RDM Steps in This Case Study

Decision framing: The CRBS included deep engagement with Colorado River Basin stakeholders to specify the uncertainties, portfolios of management strategies, models, and performance metrics used in the study. These inputs were developed iteratively over the course of the study and, in some cases, as a result of working through the steps of the RDM process (e.g., the identification of new strategies in the new futures and strategies step). Over the course of the study’s three years, dozens of meetings were held.
Evaluate strategies across futures: Using the inputs from the decision framing step, the CRBS team developed a very large set of plausible futures and, in the first iteration of the RDM process, evaluated how well the current management approach would perform through 2060 in these futures using a detailed simulation model of the system. A second iteration of RDM that included new strategies identified in the new futures and strategies step evaluated portfolios of additional management actions.
Vulnerability analysis: An RDM-style vulnerability analysis was used to analyze how well the current management approach would perform using the results of the thousands of simulations modeled in the previous step. This process identified two simple decision-relevant scenarios that describe future conditions that would require significant management changes, investments, or both.
New futures and strategies: After identifying the key vulnerabilities of the current management approach, the stakeholders developed four portfolios of additional water management actions that could reduce the identified vulnerabilities. These portfolios were then evaluated across the original set of plausible futures, and the results were analyzed as part of the trade-off analysis. Because the initial RDM analysis looked at only one strategy—i.e., the current management approach—this step preceded the trade-off analysis.
Trade-off analysis: RDM was used to showcase trade-offs in how the portfolios of water management actions improved the performance of the water management system across all futures and in the two scenarios identified in the vulnerability analysis and at what cost.
Robust strategies: The CRBS team did not try to select a single robust strategy. Instead, they focused on better understanding what new options would be required under the different vulnerabilities, identified what the costs would be, and identified key near-term options to evaluate with stakeholders in more depth.

RDM Step: Decision Framing

The CRBS process included many engagements with stakeholders representing the seven Colorado River Basin states and other interests, such as environmental, recreational, and cultural nongovernmental organizations (NGOs); specific water agencies; and tribal groups. Through ongoing conversations, the study team developed the scope of analysis, which we summarize in Table 1 using an XLRM matrix.

Table 1. XLRM Matrix for Colorado River Basin Case Study

(X) Uncertainties	(L) Management Options and Strategies
Climate drivers Demand for Colorado River water Reservoir operations post 2026	Current management^a Four portfolios composed of individual options^b Demand reduction Supply augmentation
(R) Relationships or Systems Model	(M) Performance Metrics^c
Colorado River Simulation System (CRSS)	Upper Basin reliability – Lee Ferry deficit Lower Basin reliability – Lake Mead pool elevation Cost of option implementation

^a This was analyzed in the first iteration of the decision framing and evaluate strategies across futures steps.

^b Four portfolios were developed for the CRBS in the new futures and strategies step and analyzed in subsequent iterations of the evaluate strategies across futures and vulnerability analysis steps. In a subsequent RAND/LLNL workshop, five more portfolios were developed (we describe these later in the case study).

^c A larger set of performance metrics was also evaluated and described in the CRBS, but those metrics were not included in the CRBS RDM analysis.

Uncertainties (X)

As shown in the upper-left corner of the matrix, the CRBS planners and stakeholders were concerned with three main drivers of uncertain future conditions: climate, as expressed through water runoff into the Colorado River system; demand for Colorado River supplies by the seven basin states; and system operations after the Interim Guidelines expire in 2026. To reflect this uncertainty, the CRBS developed three sets of projections, one for each driver, which were then combined to specify unique plausible futures.

Climate

To account for climate uncertainty, the CRBS used three sources of plausible climate information to devise supply projections: the recent historical record (from 1906 to 2007), the paleo record (past 1,250 years), and downscaled climate simulations (from 2000 to 2060) from global climate models (GCMs) evaluated under one of three projections of future global greenhouse gas emissions developed by the global scientific community. Together, these sources (including a blend of paleo and historical sources) supported the development of 1,959 different time series projections of future hydrological conditions. See Table 2 and the CRBS Technical Documentation for details on how the specific traces were developed.

Table 2. Supply Projection Descriptions

Supply Source(s)	Number of Projections	Description
Historical	103	Future hydrologic conditions resemble those from the past century
Paleo	1,244	Future hydrologic conditions are represented by reconstructions of streamflow for the past 1,250 years
Paleo/historical blend	500	Future hydrologic conditions are represented by a blend of the wet-dry states of the longer paleo-reconstructed record and historical magnitudes of wet and dry periods
Projected future climate conditions	112	Future hydrologic conditions are consistent with regional precipitation and temperature trends represented by an ensemble of GCMs evaluated for three global greenhouse gas emissions scenarios

Like many other RDM studies, we used interactive visualization software (in this case, Tableau) to convey information about the analysis to stakeholders and decisionmakers. For example, Visualization 1 summarizes projected temperature and precipitation trends from 41 global climate models evaluated for up to three global greenhouse gas emissions scenarios (for a total of 112 projections). Click on the top tabs to navigate through the visualization and review instructions. By exploring this visualization, you can see that there is a wide range of plausible temperature and precipitation trends, according to the models. There are large differences not only between models, which represent uncertainty in how the climate system is modeled and what initial conditions exist, but also for the same models under different assumptions of global greenhouse gas emissions. In summary, the projections show warming between about 0.75 and 4 degrees Fahrenheit by 2060. Precipitation changes range from about + or – 17 percent. Such a wide range of future conditions is challenging for long-term water resources planning because different strategies likely would be needed for different climate trends within these ranges. It is also important to note that future climate conditions could change in ways that are not captured by the GCMs.

Visualization 1. Projected Colorado River Basin Temperature and Precipitation Trends from Global Climate Models

Demand

To account for different trajectories of water use by the seven basin states, six demand projections were developed (see Table 3). Each projection reflects different assumptions about economic growth, development patterns, and environmental values, which were identified as critical issues during consultation with stakeholders. All projections show an increase in demand through 2060, which would exacerbate the supply-and-demand imbalance, with the largest increase being 118 percent for the rapid growth (variant 1) projection and the lowest increase being 105 percent for the slow growth projection.

Table 3. Demand Scenario Storylines, Scenarios, Descriptions, and Statistics

Demand Projection	Description	Average Demand, 2041–2060 (percentage of 2012 baseline)
Current projected	Continuation of long-term trends in growth, development patterns, and institutions	14.4 MAF per year (109%)
Slow growth	Slow growth with emphasis on economic efficiency	13.8 MAF per year (105%)
Rapid growth (2 variants)	Economic resurgence (population and energy) and current preferences toward human and environmental values	15.6 MAF per year (118%) [variant 1] 14.7 maf per year (111%) [variant 2]
Enhanced environment (2 variants)	Expanded environmental awareness and stewardship with growing economy	14.1 MAF per year (107%) [variant 1] 15.0 MAF per year (114%) [variant 2]

Reservoir Operations

Given the uncertainty about how the system will be operated after 2026, when the Interim Guidelines expire, two assumptions were defined. In the first assumption, the Interim Guidelines and the rules for operating Lake Powell and Lake Mead remained in effect after 2026. In the second, operations returned to pre–Interim Guidelines rules after 2026.

Management Options and Strategies (L)

In the initial iteration of the RDM process (specifically, in the evaluate strategies across futures and vulnerability analysis steps of the RDM framework), the study team analyzed the Colorado River Basin’s current management strategy. Using the results of the vulnerability analysis step, the team developed and analyzed alternative strategies focused on supply-augmentation and demand-reduction options that could improve system performance and reduce vulnerabilities. Input on the options was solicited from the public, who proposed about 150 different water management actions (Reclamation, 2012b). The team evaluated a smaller set of these actions—about 80—according to rough estimates of supply or demand reduction; availability; cost; and 16 other criteria, including technical feasibility, permitting risk, legal risk, policy risk, and energy intensity. Each action was scored according to these criteria.

The team then developed an interactive tool to help stakeholders create portfolios of water management actions from among the 80 actions that were evaluated and scored. The tool showed the users the full set of available options and the characteristics of these options. The tool then ranked the options by their levelized costs—a rough estimate of the unit cost of new supply or demand reduction. Next, the user could exclude actions from a portfolio based on the qualitative scoring criteria of each action. The tool then summarized the total cost and approximate yield (supply or demand reduction) of the remaining options and ordered them from least expensive to most expensive. As we describe in more detail below, these portfolios were modeled to be adaptive—meaning that options would be implemented during a specific simulation only if the modeled conditions warranted them.

Using this tool, the CRBS study team and stakeholders developed four portfolios for the CRBS to reflect four underlying strategies—Portfolio A (Inclusive), Portfolio B (Reliability Focus), Portfolio C (Environmental Performance Focus), and Portfolio D (Common Options). Portfolio B included actions that are viewed as well understood and time-tested and that will provide reliable supplies if implemented. Portfolio C was less restrictive in terms of long-term viability but more restrictive in terms of most environmental and social criteria. Portfolio A includes all options in Portfolio B and Portfolio C. Alternatively, Portfolio D included only those options common to both Portfolios B and C.

In a workshop organized by RAND and LLNL researchers after the completion of the CRBS, stakeholders role-played the perspectives of stakeholders in the Upper Basin, the Lower Basin, Southern California, Mexico, and the NGO community to develop five additional portfolios.

Relationships or Systems Model (R)

The CRBS team used the CRSS, a model developed in the RiverWare modeling platform, to evaluate how the Colorado River system would perform through 2060 in different futures and under different management strategies. CRSS is actively developed and maintained by Reclamation engineers at the Center for Advanced Decision Support for Water and Environmental Systems (CADSWES) at the University of Colorado Boulder. The model was updated as part of the CRBS process to evaluate the portfolios as adaptive strategies. See Chapter Three of Groves et al., 2013, for more information about this approach.

Performance Metrics (M)

The CRBS team evaluated the performance of the system using a large set of system-reliability metrics corresponding to six resource categories: water deliveries (nine metrics), electric power resources (two metrics in three locations), water quality (one metric in 20 locations), flood control (three metrics in ten locations), recreational resources (two metrics in 13 locations), and ecological resources (five metrics in 34 locations). The RDM analysis conducted for the CRBS, however, focused on two key aspects of system performance: reliability of the Upper Basin and reliability of the Lower Basin. For the Upper Basin, if the ten-year average flow at Lee Ferry (the dividing point between the Upper and Lower Basins) falls below 75 MAF, a deficit is declared, which triggers supply cutbacks to Upper Basin users. Thus, the frequency of a “Lee Ferry Deficit” is indicative of Upper Basin reliability. For the Lower Basin, the elevation of Lake Mead is a useful proxy because levels below specific thresholds trigger supply cutbacks to Lower Basin users. The team also considered the costs of additional management options in the analysis.

RDM Step: Evaluate Strategies Across Futures

This step of the RDM framework evaluates strategies across futures. In the case of the CRBS, the initial goal was to see how the Colorado River Basin’s current management strategy would perform across the plausible futures based on the uncertainties identified in the decision framing step. In that step, each of the three sets of projections (climate drivers, demand, and reservoir operations) were identified independently; however, they interact to form a complete set of plausible futures that totals 23,508 (see Figure 3).

Figure 3. Combining Uncertainties to Generate Plausible Futures

1,959Climate Scenarios multiplied by 6 Demand Scenarios multiplied by 2 Reservoir operationsScenarios equals 23,508 Plausible futures

Tools Support the Evaluation of Strategies Across Many Futures

RDM requires the use of tools for evaluating such large experimental designs and interpreting the results. RDM studies use scripts to automate (1) the specification of model inputs for a specific future and strategy, (2) the execution of the model (in this case, CRSS) for each future, and (3) the storage of key model outputs of interest in a database. These scripts can be configured to perform many simulations simultaneously across a computer cluster (as was done for the CRBS), a supercomputer (as demonstrated by RAND and LLNL researchers), or in the cloud (as was done in a recent research study using a RiverWare model and in the Metropolitan Water District of Southern California Case Study). A common approach to defining simulations is to create an experimental_design file that includes the information to specify for each simulation. For CRBS, this file included the climate projection, demand scenario, and reservoir operations specifications for each of the 23,508 futures. To illustrate how such a file is constructed, we provide a subset of the full CRBS experimental design file corresponding to only the 112 projected climate futures (see Table 2), combined with six demand projections and two reservoir operations assumptions. The design file is available for download here:

Download this data file [CSV]

For each CRSS model simulation, information about the inputs defining the future conditions and management strategy was saved, along with a subset of results that show how well a strategy performed (per the established performance metrics of reliability of the Upper Basin and reliability of the Lower Basin). For example, information about each climate specification was included in a data file called the climate_scenario_attribute file. A results_annual file and results_aggregate files included the results from each simulation recorded annually and aggregated over time, respectively. This information was saved in simple comma-separated files (.csv) for ease of use by the many team members, although, in other studies, a more complex database could be used.

Exploring How Colorado River Management Strategies Perform Across Plausible Futures

To explore how Colorado River management strategies would perform across plausible futures, we use interactive visualizations to show CRSS results for different performance metrics over time. For example, Visualization 2 shows one important metric—Lake Mead elevation over time—from the evaluation of the current management strategy. The visualization shows a subset of the 23,508 futures defined in the experimental design file, specifically each of the 112 GCM projections combined with a single demand scenario. Click through the tabs to see (1) the historical Lake Mead elevations (black line), (2) three of the 112 projections (those from the inmcm3_0.1 GCM), (3) the full set of projections corresponding to the 112 GCM climate projections, and (4) results pertaining to those GCM projections that exhibit sharply negative precipitation trends. The visualization shows that many futures would lead Lake Mead surface elevation to drop below the 1,000-foot level. Interact with the visualization to see the relationship between climate trends and Lake Mead elevation projections. Selecting results that are on the high end of the temperature range or the low end of the precipitation range (using the slider bars or by highlighting symbols in the scatter plot in the lower left) shows that both of these characterizations of future climate are important drivers of Lake Mead outcomes.

Visualization 2. Projections of Lake Mead Elevation over Time

How Did Evaluating Strategies Across Futures Using RDM Help Water Planning and Management?

Evaluating a large ensemble of futures provided the CRBS participants with a good sense of what the plausible range of future system conditions could be, from basin sustainability to collapse of the system with the emptying of Lake Mead. It also provided the data to understand what the key drivers to poor performance might be. This is explored further in the vulnerability analysis step of RDM.

This approach to addressing deep uncertainty did require significant computing resources. At the time of the study, it took two clusters of ten to 20 computers each many weeks to complete the simulations. With the availability of cloud computing resources, however, analyses of similar complexity can now be done in hours. This approach is not particularly feasible if the simulation results require manual postprocessing or quality control. For the CRBS, all postprocessing was automated and integrated into the scripts that generated the ensemble.

RDM Step: Vulnerability Analysis

It is tempting to try to assign probabilities to the different futures evaluated in a long-term study. By doing so, one can reduce uncertainty down to a single probabilistically weighted result, which can be used to compare the costs and benefits of alternative strategies. The challenge, however, is to credibly define weights for these futures. Different heuristics can be developed to do so, but, because deeply uncertain factors are by definition novel and not understandable by historical experience or testing, any set of weights is only suggestive. Different weights could be justified that would lead to a different outcome.

Interpreting thousands of different plausible futures requires a new approach to decision analysis. RDM performs a vulnerability analysis to understand which uncertainties lead to poor outcomes and would therefore necessitate an alternative strategy. The process identifies thresholds to define vulnerabilities that are not dependent on defining probabilities. The thresholds are instead defined based on current understanding of how the system would perform relative to outcomes of interest. And the definitions of vulnerabilities through a scenario-discovery process allow researchers the flexibility to define scenarios—or stories of vulnerabilities—that are compelling and useful to planners. Defining critical vulnerabilities through scenarios supports the comparison of options or strategies in a probabilistic-neutral way and provides an opportunity to consider the likelihoods of vulnerabilities as a final step in a decisionmaking process. We discuss the elements of the vulnerability analysis process in more detail below.

Vulnerability Analysis Key Terms

Vulnerability: is a general term for the uncertain conditions in which a strategy performs poorly or leads to an unacceptable outcome.
Decision-relevant scenario: is a technical term for the definition of a vulnerability, usually the output of the scenario-discovery technique.
Vulnerability threshold: is a value for a performance metric that distinguishes between acceptable and unacceptable outcomes or good and poor performance.

Defining Unacceptable Outcomes

To define a vulnerability, an analyst (often informed by stakeholders) must first define what constitutes outcomes that are unacceptable or performance that is not consistent with meeting a strategy’s goals. Vulnerability thresholds are thus defined to distinguish among acceptable and unacceptable outcomes. In some contexts, this is straightforward, because clear criteria exist that must be met. In other contexts, particularly when considering conditions that are novel, it is less clear how to establish a threshold.

For the CRBS, the study participants specified that the threshold of Lake Mead dropping below 1,000 feet during any single year would define one important set of unacceptable outcomes because the system would not adequately serve the Lower Basin in these situations. This is because 1,000 feet is the level at which withdrawals from Lake Mead to serve Las Vegas would no longer be possible under the conditions of the time of the study. Other thresholds could be defined based on different stakeholder considerations, although this would generally affect the number of futures deemed unacceptable. Setting a vulnerability threshold (for example, a Lake Mead elevation of 1,075 feet—an elevation that triggers supply curtailments under the Interim Guidelines) would result in more futures being identified as unacceptable.

Visualization 3 illustrates how the setting of a Lake Mead elevation vulnerability threshold defines how many futures would be acceptable and unacceptable. Note that as the threshold is changed, the percentage of cases that are unacceptable adjusts accordingly.

Visualization 3. Defining Acceptable and Unacceptable Minimum Lake Mead Elevation Outcomes

Using Decision-Relevant Scenarios to Identify When Alternative Strategies Are Needed

Decision-relevant vulnerability scenarios are used to concisely describe the conditions in which a candidate strategy would not perform well and, therefore, when alternative strategies must be developed to address unacceptable performance. A decision-relevant scenario, which is defined by a small number of uncertainties and their ranges, is useful when it (1) describes conditions that lead to many of the unacceptable outcomes in the ensemble, (2) describes a high percentage of unacceptable cases, and (3) is interpretable, meaning that it has some relevance to conditions that can be understood and monitored over time. RDM uses scenario-discovery techniques to define such decision-relevant scenarios by determining which uncertain factors lead to the unacceptable outcomes, as defined by the vulnerability threshold.

An Example of Scenario Discovery Using Two Variables

Recall that in the CRBS case, uncertain factors could be temperature or precipitation trends, future demand, or the operating rules post 2026. To illustrate the process of scenario discovery, Visualization 4 again plots the temperature and precipitation trends for a subset of results corresponding to the GCM climate projections, but, this time, it indicates by the color of the symbol whether the conditions lead Lake Mead elevation to drop below the threshold (initially set at 1,000 feet) for the current management approach. In the 1,000-foot threshold scenario, 80 percent of cases are unacceptable and 20 percent are acceptable. However, now the user can see how the temperature and precipitation trend corresponding to each result correlates to poor performance. Cases that have large negative precipitation trends (toward the left of the figure) and high positive temperature trends (toward the top of the figure) are more often unacceptable.

Visualization 4. Defining a Lake Mead Vulnerability Based on Future Temperature and Precipitation Trends

The scenario-discovery process next seeks to define a decision-relevant scenario (or vulnerability) by restricting the ranges of the input variables in a way that focuses on the unacceptable outcomes (the red circles in Visualization 4). For example, by setting the range of the precipitation trends in the visualization to be no more than –2 percent and the temperature trend to be more than 1.5 degrees Fahrenheit, one defines a scenario that includes 30 percent of all cases—100 percent of which are unacceptable. If these climate conditions prevail (coupled with the demand and assumptions in the Interim Guidelines), then the analysis suggests a very high likelihood that Lake Mead elevation would drop below 1,000 feet. You can create your own definition of a decision-relevant scenario (or vulnerability) with these two variables using Visualization 4. A perfectly defined vulnerability scenario would include 100 percent of the cases in which Lake Mead elevation drops below 1,000 feet and 0 percent of the cases in which it does not. Such a scenario is not possible to find with only these two variables.

Scenario Discovery for the CRBS Analysis

For the CRBS analysis, the study team decided to define vulnerabilities based not on the climate trends corresponding to the supply scenarios but instead on the specific features of Colorado River flow, which is more-easily observed. There is no objective and analytic method for identifying definition parameters; rather, this process is the result of experimentation and expert judgment. However, there are several guiding principles. First, the parameters used to characterize vulnerabilities should be measurable over time. For the CRBS, the parameters included are easily measurable because they are based on aggregate water demands, river flows at specific locations, and basin state negotiations regarding system operations. This ensures the relevance of these parameters for designing a robust, adaptive strategy. Second, the set of parameters should reflect the uncertainties in the experimental design in some way. Specifically, the CRBS analysis considered the parameters shown in Table 4 related to the three main uncertainties.

Table 4. Uncertainties Used in Scenario Discovery Analysis

Uncertainty	Related Parameter
Water demand	Total demand Upper Basin demand (average) Lower Basin demand (average)
Water supply	Lake Powell mean inflow Lake Powell minimum 5-year-mean flow Lake Powell minimum 8-year mean flow Lake Powell minimum 10-year mean flow Lake Powell minimum 15-year mean flow Lee Ferry mean flow Lee Ferry minimum 5-year mean flow Lee Ferry minimum 8-year mean flow Lee Ferry minimum 10-year mean flow Lee Ferry minimum 15-year mean flow
System operations	Extension of Interim Guidelines after 2026

When the uncertainties are more extensive than the two-variable temperature and precipitation trends, for example, a more sophisticated approach called the Patient Rule Induction Method (PRIM) can be used to define vulnerable scenarios. PRIM was first used in a DMDU study to identify the vulnerabilities of long-term policies. This specific approach was later more formally described and compared with other methods and continues to be widely used because of ease of use and freely available software in R and Python and because it defines easily interpretable scenarios. PRIM and other analytic scenario-discovery tools are used to (1) identify which uncertainties or characterizations of uncertainty are most important in determining future conditions to which a system is vulnerable and (2) define a concise set of rules that describe the range of uncertainty. These tools are most valuable when there is a large number of evaluations of the future to analyze.

Specifically, PRIM iteratively identifies “boxes,” which are defined by the included range of different input variables that balance the following three characteristics (or measures of merit):

coverage: the percentage of all futures in the vulnerable scenario in which conditions are not acceptable. Ideally, the vulnerability would contain all such cases in the database and coverage would be 100 percent.
density: the percentage of all of the futures in the vulnerability that are unacceptable. Ideally, all of the cases within the vulnerability would be unacceptable and density would be 100 percent.
interpretability: the ease with which users can understand the information conveyed by the vulnerability. The number of uncertainties used to define the scenario serves as a proxy for interpretability. In general, the smaller the number of uncertainties used, the higher the interpretability is.

For the CRBS, PRIM was used to identify the key vulnerabilities of the Upper Basin and the Lower Basin. We focus just on Lower Basin vulnerabilities for this case study.

You can view the PRIM tool to see how it identifies key vulnerabilities using a subset of the parameters and simulation results from the CRBS study.

View PRIM Tool

There are other algorithms that can be used to support scenario discovery. Dimensional stacking, for example, is described in Chapter 7 of Marchau et al., 2019, and the C5.0 algorithm is used in the Monterrey, Mexico, Case Study to define adaptive strategies based on the identified vulnerabilities.

RDM Step: New Futures and Strategies

After identifying both Lower Basin and Upper Basin vulnerabilities, the CRBS team worked with stakeholders to develop four portfolios of additional water management options to reduce these vulnerabilities. As described in the Management Options and Strategies (L) section, an interactive tool was developed that enabled stakeholders to include options in a portfolio that had desired attributes related to non–supply and demand issues, such as energy use, reliability, and regulatory burden. These portfolios were then modeled in CRSS dynamically, such that options would be implemented only when the simulated conditions were suggestive of the vulnerable conditions. Each simulation not only estimated performance with respect to Lake Mead elevations and the other performance metrics but also evaluated which of the portfolio options would be needed, when they would be needed, and at what cost. In general, more-stressing futures require more of the options in a particular portfolio and thus lead to higher costs; under a more favorable future, fewer new options would be needed.

After the completion of the CRBS, RAND and LLNL researchers simulated a stakeholder engagement meeting in which a new set of portfolios was developed using the same CRBS portfolio tool during the first part of a workshop. The following five portfolios were developed and named based on the stakeholder role played by the workshop participants:

Upper Basin includes a broad range of lower-cost conservation and supply-augmenting options to improve the supply-and-demand imbalance and preserve Upper Basin reliability
Lower Basin includes more-expensive options to improve the supply-and-demand balance; also includes options to offset Southern California demand
Southern California includes more-expensive near-term options to reduce demand and increase supply; also includes longer-term investments in water reuse and local supply augmentation
Mexico includes a wide variety of options to improve the supply-and-demand balance
NGOs includes options that seek to maintain river flows and support ecosystems and recreation.

Two of the four CRBS portfolios were also included in the set of portfolios—Common Options and Inclusive. As in the CRBS, these portfolios were then evaluated across the original set of approximately 23,500 futures and analyzed with respect to Upper and Lower Basin performance metrics (the evaluate strategies across futures and vulnerability analysis steps of RDM).

For the CRBS, the simulations took many weeks. For the RAND-LLNL workshop, high-performance computing facilities (i.e., a supercomputer) performed the same calculations during lunch.

RDM Step: Trade-Off Analysis

For both the CRBS and the RAND-LLNL workshop, evaluating the new strategies across all of the futures and comparing them with current conditions revealed robustness and cost trade-offs. In general, the fewer the futures in which the strategy is performing poorly, the more robust the strategy is. Comparing this metric of robustness with the cost of implementing the strategy provides an actionable trade-off from which to make decisions.

The upper-left graphic in Visualization 5 shows this trade-off for the seven portfolios evaluated in the RAND-LLNL demonstration. A result in the lower-left corner would be preferred because it would have low vulnerability (i.e., a high number of acceptable outcomes) and low cost. The range of costs shown for each portfolio accounts for the variation in options that are implemented for each future. Accompanying the trade-off graphic is another graphic that shows how frequently different options are implemented across the futures as a measure of how likely it is that the options will be needed and how long of a delay there is before the options are needed. The two graphics are linked, so the user can focus on the results for a specific portfolio or type of option across the two graphs. By exploring this visualization, one can see how the inclusion of different options can reduce vulnerability and what the effect would be on cost. The second tab includes all seven portfolios.

Visualization 5. Colorado River Basin Portfolio Outcomes and Trade-Offs

NOTES: M&I = municipal and industrial; NA = not applicable.

The trade-off analysis step of RDM accomplishes two key things. First, it provides a way of evaluating the performance of strategies across uncertainties by reflecting vulnerability reduction and costs. Next, by examining which options are included in the portfolio and when, stakeholders can define an adaptive strategy. For example, by examining the results presented above, the Colorado River Basin states could choose the Lower Basin strategy; it reduced the vulnerability the most and does not cost as much as the Highly Inclusive or Southern California portfolios. Implementing this strategy in an adaptive way could involve the following steps:

Implement options that are frequently needed in the near term, such as
1. municipal and industrial (M&I) conservation
2. agricultural conservation
3. municipal reuse projects
Prepare for other options that are frequently needed in the longer term, such as
1. desalination in Yuma
2. desalination of specific groundwater basins
Monitor conditions over the next five years and begin more-costly options if needed, such as
1. desalination on the Salton Sea
2. industrial and gray water reuse.

Although the CRBS did not select a portfolio, additional workgroups were set up as part of Reclamation’s Moving Forward initiative to further study and work toward the implementation of the top options included in several portfolios: M&I Conservation, Agricultural Conservation, and Municipal Reuse.

Case study authors: David Groves and James Syme

Acknowledgments: The authors would like to thank Carly Jerla, Alan Butler, and James Prairie of CADSWES and Rebecca Smith of Reclamation for their review of this case study and helpful suggestions.