A recurring challenge to successful acquisition program execution is poor contractor performance. Air Force officials asked RAND Corporation researchers to develop and prototype data-analytic methods on a variety of government and commercial data sources to assess relative contractor performance risks in Air Force acquisition contracts and programs. They present initial insights and approaches in exploring and prototyping these methods.

Cover: Relative Contractor Risks
Read Online

Relative Contractor Risks

A Data-Analytic Approach to Early Identification

by Philip S. Anton, William Shelton, James Ryseff, Stephen B. Joplin, Megan McKernan, Chad J. R. Ohlandt, Samantha Cohen

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 1.8 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Purchase

Purchase Print Copy

 FormatList Price Price
Add to Cart Paperback94 pages $29.50 $23.60 20% Web Discount

Research Questions

  1. What techniques can be employed to identify contractor performance risk indicators relative to their peers?
  2. How can program managers and stakeholders leverage available government and commercial data to identify these risk indicators?
  3. What data sources are available and useful?
  4. What practical concerns arise when implementing such risk assessments?

Risk is a key component in any business transaction, especially transactions worth millions of dollars or more. As the Department of the Air Force acquisition professionals are aware, there are multiple kinds of risks in developing and acquiring new systems. One recurring challenge to successful acquisition program execution is poor contractor performance. When contractors are in danger of not meeting contractual performance goals, acquisition may not be fully aware of the shortfall until, for example, a schedule deadline is missed, government testing indicates poor performance, or costs exceed expectations.

The authors developed and prototyped a new way to apply data analysis on a variety of government and commercial data sources to assess the relative contractor performance risks in Air Force acquisition contracts and programs. This method produces risk indicators earlier than do current information sources and metrics by analyzing workforce, costs, financial health, influence, supply chains, past performance, and other data to assess relative risk indicators.

Such automated tools can help managers focus their limited resources on potential risks buried in large, diverse data and take mitigation actions based on program relevance and impact on desired risk levels. Identified outliers are indicators for acquisition professionals to apply their acumen, understanding of program priorities, and acceptable levels of risk to determine the relevancy and magnitude of the potential risks and what actions should be taken (if any) to mitigate them.

Key Findings

Data analytic tools can help acquisition managers identify potential contractor risks indicators early on

  • Statistically comparing contractor outliers relative to peers appears to be a useful way to objectively identify potential risks; this approach identifies areas for increased management attention.
  • Cross-indexing public and sensitive databases through modern interfaces enables new risk indicators too time-consuming to discover manually.
  • Such automated tools can help managers focus their limited resources on potential risks buried in large, diverse data and take mitigation actions based on effectiveness and program relevance.
  • Identified outliers are indicators for acquisition professionals to apply their acumen, understanding of program priorities, and acceptable levels of risk to determine the relevancy and magnitude of the potential risks and what actions should be taken (if any) to mitigate them.

Some challenges remain to operationalize this capability

  • Some data that are important for assessing relative contractor risks are very difficult to obtain — even for Air Force officials and federally funded research and development centers, let alone support contractors.
  • Further work is necessary in developing a prototype with significant critical mass of data sources and measures to test and refine this approach. User feedback on utility and design is also needed.

Recommendations

  • Integrating and assessing traditional and nontraditional data sources provides indications of potential areas of concern.
  • Air Force program managers and stakeholders should use this approach to identify relative risks for further due diligence, confirmation, and proactive management.
  • Additional data and the inclusion of more metrics are needed to make this more robust; data availability, accessibility, and analysis are key.
  • This is a research prototype and is not ready for transition to an operational system. Despite its limitations, however, this approach is more sophisticated in some ways than other available systems and may point to features or concepts that could be added to Air Force or Department of Defense systems that assess potential contractor risks.

Table of Contents

  • Chapter One

    The Challenge: Contractor Risks in Acquisition

  • Chapter Two

    Taxonomy of Relative Risks

  • Chapter Three

    Relative Risk Measures and Equations

  • Chapter Four

    Prototype Architecture

  • Chapter Five

    Insights, Conclusions, and Next Steps

  • Appendix A

    Risk Measures and Potential Data Sources

Research conducted by

This research was commissioned by the Deputy Assistant Secretary for Acquisition Integration (SAF/AQX), Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics and conducted by the Resource Management Program of RAND Project AIR FORCE.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.