Choosing Defense Project Portfolios

The U.S. Budget Control Act of 2011 presents the U.S. Army and the rest of the Department of Defense (DoD) with unprecedented fiscal challenges. Austerity will be the watchword, while the need for mission-capable weapon systems will continue. Army and other DoD decisionmakers thus face increased urgency in their attempts to reap savings through improved efficiency and cost-effectiveness. Among those attempts, they will have to follow a 2006 directive from the Deputy Secretary of Defense that all DoD agencies use capability portfolio management to optimize investments and minimize risk in meeting needs across the defense enterprise.

With these needs and guidance in mind, the RAND Arroyo Center in 2006 began developing a methodology for selecting and managing portfolios of projects in the acquisition process. The methodology, currently available only at RAND, has several distinctive features:

• It permits choosing, from billions or even trillions of possible project portfolios, those that best meet the requirements, the budget, or some other constraint.
• It accounts for overlaps and redundancies across projects; that is, it does not simply add to the portfolio what traditional methods find as the next most cost-effective project under consideration.
• It accounts for uncertainties, e.g., in the budget and the weapon cost, and adjusts the portfolio to hedge against such uncertainties.

RAND's portfolio analysis and management method, PortMan, which includes a mixed integer programming model and a simulation, could find application to a broad array of projects in a wide variety of government and business portfolios. The most extensive demonstration of the method's capabilities to date has been in the area of defense system acquisition, where it has been applied to science and technology (S&T) projects, as well as those in the engineering and manufacturing development (EMD) stage of the acquisition process. Sponsored by the Deputy Assistant Secretary of the Army for Cost and Economics, PortMan was designed to help the Army identify optimal investments in effective and affordable weapon systems. This brief describes the steps taken in applying RAND's methodology to acquisition planning and shows how PortMan's features, listed above, are exploited in the process.

In a nutshell, the RAND approach to portfolio management can aid Army S&T and acquisition planners in two key planning tasks: first, managing the supply of and demand for defense systems, and, second, identifying the optimal total remaining lifecycle budget of a project portfolio, as well as the optimal shares for S&T and EMD, and the optimal portfolio at that budget (or, in the event of budget-cutting, an optimal S&T/EMD portfolio at a suboptimal budget).

Mapping Supply and Demand

The first step in the RAND framework is to broadly identify where the Army may encounter problems meeting requirements with the portfolio of S&T/EMD projects currently proposed or already in development. This broad map of supply and demand is then refined, considering that some projects in this portfolio (or any other) will not lead to fielded systems, either because projects fail or some system turns out to be less cost-effective than others.

Specifically, the RAND approach begins with a large matrix or table that maps requirements (actually, requirement gaps—as determined, e.g., from U.S. Army Training and Doctrine Command data) against projects intended to help fill one or more of them. Analytic methods and expert judgment are brought to bear to determine the extent to which projects will fill requirements.^[1] Each requirement (or requirement category), then, will be filled to some degree by the portfolio of projects. Supposing for now that projects do not fail, the graph might look like that shown in Figure 1, where many of the Army's S&T requirement categories are redundantly met by the current portfolio of projects. An exception is requirement category 10, which is judged to be only 57 percent filled by the current portfolio of projects;^[2] three others are less than 200 percent filled.^[3] On the whole, though, the current project portfolio meets requirements well—indeed, the graph suggests that there are opportunities for taking funds from projects that mainly address requirements that will already be met by other current projects and redirecting those funds to projects not yet in the current portfolio. The latter could be selected to help fill partially met requirements, such as requirement 10, and requirements that have a safety margin inadequate to deal with adverse future uncertainties, such as project failures and cost overruns.

Now, can the current portfolio meet the same set of requirements to the same fill levels (57 percent of requirement category 10 and 100 percent of all others) if some projects are expected to fail, as is likely? The researchers assumed a failure rate of 10 percent across all projects^[4] and ran a simulation in which each project takes a random draw with a 10 percent chance of failure. After 10,000 simulation runs, the chance that the current portfolio would be able to successfully fill the same set of requirements to the same levels is estimated to be only 16 percent. This underscores the importance of taking the probability of failure into account.

The next step was to determine whether the percentage of success could be raised substantially by modestly lowering the target fill levels of some requirements and thus the success criterion. The answer to this question was yes: By using the PortMan tool, the researchers found that, for the same project failure rate, the probability of meeting the target set of requirements could be raised to 73 percent if the requirement 10 target fill level was dropped to 36 percent and the requirement 6 target to 88 percent (and the others kept at 100 percent). Further modest easing of target set requirements could raise the probability of meeting all targets to just about 100 percent. The RAND framework thus clarifies the tradeoffs to be made in moving from a very low level of confidence that almost all categories of requirements will be filled, to a very high level of confidence that a somewhat lower set of targets will be met. What happens to the requirement categories not fully filled in the latter case? They might be addressed by new projects, which could be funded out of the savings gained by reducing the level of redundancy in meeting other requirement categories.

Selecting the Optimal Project Portfolio and Budget

At issue is how to select projects so that together they will have the best chance to meet all categories of requirements at desired levels for a given budget. Clearly, if money were no object, one would fund all projects so as to have the highest chance to fill all requirements. As the available funds are reduced into some reasonable range, the chance of filling all requirements would be reduced. The question is by how much? PortMan can help portfolio managers answer that question.

To illustrate the tradeoff, the researchers estimated that it would take $3.1 billion to finish the Army's current portfolio of S&T projects and $135 billion to develop and field all the resulting systems. Because of the redundancy with which this portfolio meets requirements, enough projects could be deleted from the portfolio to cut the total remaining cost^[5] of $138 billion in half while filling all requirements to the same target levels. Cutting the portfolio cost about in half again to $35 billion would drop the likelihood of success at meeting the same requirement set by only 10 percentage points, to 88 percent. Further cuts, even modest ones, would exact greater decrements in the success probability (compare the right ends of the curves in Figure 2). Thus, according to the PortMan tool, an efficient path for the Army would be to spend $35 billion in total remaining cost to fund its portfolio of current projects and their future systems.

However, there are further opportunities for even greater efficiency: Within the same $35 billion, the chance to fill all requirements actually peaks at a $2 billion total remaining S&T budget. That point thus represents an optimal "sweet spot" combination of cost and success rate—that is, the combination of total remaining S&T and lifecycle costs across the portfolio and probability of filling all requirements. Thus, using PortMan, decisionmakers can pick out a portfolio of projects that will have a 91 percent likelihood of achieving the modestly reduced requirement fill targets under a constraint of $35 billion in total remaining lifecycle costs, including $2 billion in total remaining S&T costs, across that portfolio. Optimally, the Army would spend this much on the current portfolio to be most cost-effective. However, if the Army's budget is less than optimal, PortMan can still find, for that budget, the optimal portfolio and the most cost-effective way to use the (suboptimal) amount of funds.

Exploiting PortMan's Distinctive Advantages

Choosing the optimal portfolio out of billions. For N projects, the number of possible portfolios (yes or no decisions on each project) is 2^N (e.g., for N = 33 projects, 2^N = over 8 billion). PortMan's algorithms permit choosing the optimal portfolio and can do so without prohibitively large computing resources. Traditional methods are able to preselect only about ten possible portfolios for analysis and comparison of outcomes under uncertainty. The chance that the optimal portfolio is among these ten, out of typically far greater than 8 billion possible portfolios, is practically nil. Consider an analogy: Suppose that each of the world's 7 billion people had a copper key, except for one person who had a gold key. Using traditional methods of analysis to find the optimal project portfolio is like trying to find the gold key by searching the belongings of only ten persons. In contrast, PortMan in effect searches all possible portfolios. This capability is, to our knowledge, unique among portfolio management tools designed to handle uncertainties.

Getting beyond individual-project cost-effectiveness. There is another reason why the portfolio selected through the PortMan-driven analysis cannot be determined by using traditional methods. The latter might, for example, call for forming an optimal portfolio by first selecting the most cost-effective project, that is, the one whose individual contribution to filling requirements, considered alone, is the greatest relative to its remaining S&T cost. Next, the second-most cost-effective project, in those terms, is chosen, and so on until the total remaining S&T budget runs out. In Figure 3, which orders the Army's S&T projects according to this concept of cost-effectiveness—the least at the top and the most at the bottom—that method would simply select the projects from the bottom up until the budget runs out.^[6] In contrast, the green bars indicate those projects included in PortMan's optimal portfolio, and the red bars indicate those not included. Some projects that would be comparatively cost-effective if considered on their own are rejected because the requirements they satisfy are redundantly met by other projects. As a result, the "optimal" portfolio chosen by the traditional method has a much lower chance of filling the given set of requirements than the equal-cost PortMan portfolio does. Alternatively, if it is cost that is allowed to vary and the objective is to fill a set of requirements at a confidence level that the traditional method can achieve, PortMan can select a less expensive portfolio for meeting that objective.

Addressing further uncertainties. Uncertainty concerning project success is only one of the types of uncertainty that the RAND PortMan framework can accommodate. Others include uncertainties in project costs (i.e., the possibility of cost overruns) and uncertainties in the budgets available. An example was demonstrated in an analysis of data on projects in the EMD phase. For these data, portfolios could be identified that would have a very nearly 100-percent chance of achieving a target set of requirements at a sweet spot of a known $25 billion total remaining lifecycle cost and $0.7 billion total remaining research and development cost (a 100-percent certainty that no project will fail is assumed for this case). When the same analysis was run for a total remaining lifecycle cost known only to fall within a $20 billion to $25 billion window, the constraint on the total remaining S&T budget had to be raised to $1.0 billion to achieve even an 83 percent chance of meeting the requirements (a higher success rate was not possible). PortMan can thus help planners quantify how real-world uncertainties will affect their expectations of portfolio success. It can also suggest to the planner how much more of the total remaining lifecycle budget should be allocated to S&T so as to increase the probability of success in the face of uncertainty.

A New Mindset in Acquisition Management

The challenges that the Army faces in selecting projects that will meet future capability needs at an affordable overall cost are not new. Uncertainty has always complicated decisionmaking. However, traditional methods are able to consider only a tiny part of the uncertainty space, for example, via simulation of a dozen or so preselected portfolios and sensitivity analysis of the chosen "optimal" portfolio, which is highly unlikely to be the true optimum. It is also difficult to think in terms of the "big picture" regarding the full life-cycle costs of fielding a system—from basic research through operations and support—rather than the unique costs of one particular acquisition stage. Portfolio analysis aims to make "big picture" thinking under uncertainty a consistent aspect of the evaluation process. Failure to do so can lead to dramatically different, often inferior, investment selections and outcomes—for example, if the result is a false impression of how well a given portfolio will fill capability gaps. Potentially severe budget constraints also make it advisable to be able to distinguish between "must have" and "desirable" requirements. The newest version of PortMan provides a means for the Army's portfolio managers to perform these analyses and more. Being able to think in terms of overall costs, setting priorities across requirements, and bringing uncertainty into the mix can all contribute to a new acquisition management mindset. This new mindset can help the Army fulfill DoD's desire for savings from improved efficiency and effectiveness in a future that realistically will be full of unknowns.

Notes

• [1] For the applications demonstrated to date, RAND has used its own analysis and judgment. However, PortMan will function with similar utility as a decision aid when different determinations of requirement fulfillment are used.
• [2] We intentionally do not here identify requirement categories or indicate particular projects for termination of further funding. The focus here is on the capabilities of the PortMan tool and not on the specific results of the demonstration runs we undertook.
• [3] A fill rate higher than 100 percent is needed to allow for the failure of some projects.
• [4] PortMan can accept different failure rates for different projects.
• [5] Some projects are already ongoing, and past or sunk costs are not included in PortMan. The total remaining lifecycle cost of $138 billion is the sum of the total remaining S&T cost of $3.1 billion and the total implementation cost of $135 billion.
• [6] To save space, only the odd-numbered projects are shown here.