The authors argue that there are limitations to relying on empirical data to guide acquisition policy. This report summarizes the case for a broader evidence base for defense acquisition policymaking and then focuses on one specific tool that the authors suggest might add analytic value: policy gaming. It also describes a prototype game focused on Middle-Tier Acquisition that researchers developed to enrich the available evidence base.
- What are the strengths and limitations of empirical data for setting acquisition policy?
- What are alternative approaches to setting acquisition policy, and what are their strengths and weaknesses?
- How can games be useful in setting acquisition policy?
- What are the limitations of games in setting acquisition policy?
One of the primary responsibilities of the Under Secretary of Defense for Acquisition and Sustainment (USD[A&S]) is to ensure the health of the overall defense acquisition system (DAS). USD(A&S) can bolster the health of the DAS by developing and promulgating sound acquisition policy that improves the function and operation of the DAS at the enterprise level. The premise of this report is that acquisition policymaking should be data driven. However, there are limitations to relying on empirical (e.g., historical) data to guide acquisition policy. In light of these limitations, the authors argue that acquisition policymaking should be evidence based, in recognition of a wider variety of analytic tools that can be brought to bear on acquisition policy questions. This report, intended for acquisition professionals, summarizes the case for a broader evidence base and then focuses on one specific tool that the authors suggest might add analytic value: policy gaming.
Policy gaming can be used to generate observations about how stakeholders might change their decisionmaking and behavior in light of changes in policy. Because the strengths and limitations of games differ from those of traditional tools for acquisition analysis, the authors argue that games complement the existing portfolio of analytic approaches. The authors describe a prototype game focused on Middle-Tier Acquisition (MTA) policy that RAND researchers developed to enrich the available evidence base to support acquisition policymaking, summarize insights from the game, and offer several next steps for USD(A&S) to consider.
Standard approaches to data-driven acquisition policy research might not be sufficient when seeking to anticipate new, substantial policy changes
- Data on the historical performance of the DAS are not always generalizable to the new policy under exploration.
- Current conditions might not hold in the future, so an approach that focuses only on current conditions risks missing external forces that have been shown to affect the health of the DAS.
- In some cases, the data might not be sufficiently rich to permit establishment of a causal effect of acquisition policy on acquisition outcomes.
- It might be difficult to tease out the effect of acquisition policy from the effect of myriad other factors that are changing simultaneously.
Gaming, a tool used in other areas of defense policy analysis, has promise to help inform acquisition policy creation and implementation
- Games can provide useful evidence about proposed policies by providing a sandbox to observe decisionmaking.
- Games appear to be valuable in cases where relevant real-world data are not available because the new policy or other condition of interest has not yet occurred.
A prototype gaming effort conducted by RAND researchers was able to anticipate potential implementation problems for MTA policy, illustrating the potential utility of gaming as an approach to inform acquisition policymaking
- The decision to acquire a program through MTA might depend on factors beyond cost and schedule.
- The risks of transitioning programs between pathways is not well understood.
- The perceived flexibility of MTA might translate into novel acquisition strategies.
- MTA might need governance to align policy implementation across the U.S. Department of Defense.
- Look to expand beyond traditional evidence-based approaches to acquisition policy. Fund pilot efforts using new tools, in order to better understand their utility in evaluating the health of the DAS.
- Instrument the adaptive acquisition system to enable continuous, incremental improvements to acquisition policy.
- Experiment with developing and using policy games to rapidly prototype more-revolutionary changes to acquisition policies for which an empirical approach might not apply.