The authors argue that there are limitations to relying on empirical data to guide acquisition policy. This report summarizes the case for a broader evidence base for defense acquisition policymaking and then focuses on one specific tool that the authors suggest might add analytic value: policy gaming. It also describes a prototype game focused on Middle-Tier Acquisition that researchers developed to enrich the available evidence base.

Building a Broader Evidence Base for Defense Acquisition Policymaking

Elizabeth M. Bartels, Jeffrey A. Drezner, Joel B. Predd

ResearchPublished May 19, 2020

One of the primary responsibilities of the Under Secretary of Defense for Acquisition and Sustainment (USD[A&S]) is to ensure the health of the overall defense acquisition system (DAS). USD(A&S) can bolster the health of the DAS by developing and promulgating sound acquisition policy that improves the function and operation of the DAS at the enterprise level. The premise of this report is that acquisition policymaking should be data driven. However, there are limitations to relying on empirical (e.g., historical) data to guide acquisition policy. In light of these limitations, the authors argue that acquisition policymaking should be evidence based, in recognition of a wider variety of analytic tools that can be brought to bear on acquisition policy questions. This report, intended for acquisition professionals, summarizes the case for a broader evidence base and then focuses on one specific tool that the authors suggest might add analytic value: policy gaming.

Policy gaming can be used to generate observations about how stakeholders might change their decisionmaking and behavior in light of changes in policy. Because the strengths and limitations of games differ from those of traditional tools for acquisition analysis, the authors argue that games complement the existing portfolio of analytic approaches. The authors describe a prototype game focused on Middle-Tier Acquisition (MTA) policy that RAND researchers developed to enrich the available evidence base to support acquisition policymaking, summarize insights from the game, and offer several next steps for USD(A&S) to consider.

Key Findings

Standard approaches to data-driven acquisition policy research might not be sufficient when seeking to anticipate new, substantial policy changes

  • Data on the historical performance of the DAS are not always generalizable to the new policy under exploration.
  • Current conditions might not hold in the future, so an approach that focuses only on current conditions risks missing external forces that have been shown to affect the health of the DAS.
  • In some cases, the data might not be sufficiently rich to permit establishment of a causal effect of acquisition policy on acquisition outcomes.
  • It might be difficult to tease out the effect of acquisition policy from the effect of myriad other factors that are changing simultaneously.

Gaming, a tool used in other areas of defense policy analysis, has promise to help inform acquisition policy creation and implementation

  • Games can provide useful evidence about proposed policies by providing a sandbox to observe decisionmaking.
  • Games appear to be valuable in cases where relevant real-world data are not available because the new policy or other condition of interest has not yet occurred.

A prototype gaming effort conducted by RAND researchers was able to anticipate potential implementation problems for MTA policy, illustrating the potential utility of gaming as an approach to inform acquisition policymaking

  • The decision to acquire a program through MTA might depend on factors beyond cost and schedule.
  • The risks of transitioning programs between pathways is not well understood.
  • The perceived flexibility of MTA might translate into novel acquisition strategies.
  • MTA might need governance to align policy implementation across the U.S. Department of Defense.

Recommendations

  • Look to expand beyond traditional evidence-based approaches to acquisition policy. Fund pilot efforts using new tools, in order to better understand their utility in evaluating the health of the DAS.
  • Instrument the adaptive acquisition system to enable continuous, incremental improvements to acquisition policy.
  • Experiment with developing and using policy games to rapidly prototype more-revolutionary changes to acquisition policies for which an empirical approach might not apply.

Topics

Document Details

Citation

RAND Style Manual
Bartels, Elizabeth M., Jeffrey A. Drezner, and Joel B. Predd, Building a Broader Evidence Base for Defense Acquisition Policymaking, RAND Corporation, RR-A202-1, 2020. As of September 23, 2024: https://www.rand.org/pubs/research_reports/RRA202-1.html
Chicago Manual of Style
Bartels, Elizabeth M., Jeffrey A. Drezner, and Joel B. Predd, Building a Broader Evidence Base for Defense Acquisition Policymaking. Santa Monica, CA: RAND Corporation, 2020. https://www.rand.org/pubs/research_reports/RRA202-1.html.
BibTeX RIS

Research conducted by

This research was sponsored by the Office of the Secretary of Defense and conducted within the Acquisition and Technology Policy Center of the RAND National Defense Research Institute.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.