Assessing Department of Defense Use of Data Analytics and Enabling Data Management to Improve Acquisition Outcomes
Aug 13, 2019
In 2016, Congress raised concerns about whether the U.S. Department of Defense (DoD) is making optimal use of data analytics in its acquisition decisionmaking. The Joint Explanatory Statement of the Committee of Conference accompanying the fiscal year (FY) 2017 National Defense Authorization Act directed the Secretary of Defense "to brief the Armed Services Committees of the Senate and House of Representatives on the use of data analysis, measurement, and other evaluation-related methods in DOD acquisition programs."
As part of this effort, the Office of the Under Secretary of Defense for Acquisition and Sustainment asked the National Defense Research Institute (NDRI), a federally funded research and development center (FFRDC) operated by the RAND Corporation, to inform the secretary's briefing to the committees. In its study, NDRI took a broad view of the role of — and support for — data analytics in defense acquisition, reaching the following conclusions:
The scope of this research was determined by the definition of the terms acquisition and data analytics, which may mean different things to different people. NDRI embraced broad definitions of both, reflecting the issues framed in the conference report and DoD parlance. In particular, NDRI adopted the definition of acquisition used by the Defense Acquisition University (DAU):
The conceptualization, initiation, design, development, test, contracting, production, deployment, integrated product support (IPS), modification, and disposal of weapons and other systems, supplies, or services (including construction) to satisfy DoD needs, intended for use in, or in support of, military missions.
Similarly, based on Congress's conference report, NDRI adopted a broad conception of data analytics for acquisition: data analysis, measurement, and other evaluation-related methods (i.e., techniques to assess and analyze data) to inform acquisition decisions, policymaking, program management, evaluation, and learning. Notably, the focus was neither on "big data" or advanced analytics nor on specific data elements or techniques the DoD should be using. Rather, the study was scoped to focus on data and analytics in their broadest sense across the acquisition system.
NDRI relied on a mixed-method approach to address the broad scope. NDRI reviewed and synthesized an array of policy, legislation, defense budgets, published literature, research findings, data on IT systems supporting acquisition, and educational institutions' course curricula. NDRI also conducted semistructured interviews with a variety of subject-matter experts throughout the DoD. NDRI used multiple analyses to measure the overall extent of DoD data analytics, including a functional decomposition and a map of data and applied analytics to acquisition functions and decisions; examinations of the availability and use of data analytics in selected major programs; quantitative analysis of budgets for the analytic workforce, major information systems, and R&D for analytic capabilities; examination of progress and trends in acquisition information and analytic systems; and assessment of the maturity of DoD efforts relative to various maturity models. NDRI assessed the DoD relative to published best practices.
This research approach embraces the breadth of congressional inquiry with limitations on the depth. NDRI did not try to assess what specific acquisition data or analytic techniques are needed. A survey (a data call) was proposed to solicit specific examples of data analytics underway in the DoD acquisition community, but it was deemed infeasible within the available time and resources and likely to produce insufficient insight. Instead, the experience, knowledge, and judgment of the authors were used to synthesize and analyze available information and fill gaps in primary data, published research, and other secondary data.
NDRI found that some manner of data analytics techniques is being applied across the whole acquisition life cycle, including market research, cost estimation, risk analysis, basic science and engineering, test and evaluation, security, supply-chain concerns, contracting, production, auditing, and sustainment. Techniques vary widely and include quantitative analysis, qualitative analysis, predefined formula and forms, systems analysis, data mining, statistical analysis, classification, clustering, outlier detection, filtering, text analytics, visual analysis, and machine learning. Data analytics contribute to major program decisions throughout the entire chain of command, from program management to acquisition executives and other stakeholders across the DoD and Congress, along with other considerations.
The DoD has implemented some aspects of data governance and management needed to enable analytics. These include strategizing and planning; establishing data requirements and use cases; authoritative sourcing; archiving, curating, and data sharing; managing security issues; working on backups and recovery; developing training and support; establishing data definitions and standards; and assessing, auditing, cleaning, transforming, and purging data. However, the maturity of these practices varies across DoD acquisition organizations.
One challenge in data management across the DoD is ensuring common data definitions to allow cross- organizational data analysis. Although some business practices provide standardization, other domains need more-active governance and management. A particular challenge is associated with the collection and use of unstructured data — that is, those that are not in fixed locations but are in free-form text, in contrast to structured data, which are easily identified and located within an electronic structure, such as a relational database.
Applications of data analytics in the acquisition environment are continuously evolving and span a range of maturity levels, from the use of simple isolated systems for archiving data about procurement to research on more-advanced analytics, such as machine learning and predictive (risk) analysis. Modern commercial off-the-shelf analytic software, such as business intelligence tools, are increasingly replacing preexisting analytic and visualization tools and dashboards.
Many data analytics capabilities have been implemented across OSD and the individual military services in recent years; these examples illustrate the trends:
Exploratory research efforts — including advanced analytics — are being pursued at the Defense Advanced Research Projects Agency (DARPA), DoD labs, FFRDCs, university-affiliated research centers, and universities.
Separately measuring the extent of analytic capabilities supporting acquisition is difficult, given that they are not accounted for as such in the DoD's workforce and operation budgets. However, NDRI developed estimates based on parametric analysis of the size of the acquisition workforce, its functions, and readily available budgetary data. This analysis suggest the DoD spends about $11–$15 billion per year on analytic workforce capabilities. The DoD also spends about $3 billion per year (about $0.5 billion for acquisition systems and about $2.5 billion for logistics and supply-chain systems) on major information systems supporting acquisition and sustainment (not desktop computing). These systems involve a mix of acquisition process support, data collection and archiving, and data analytic layers, shedding light on the resources and capabilities that ultimately inform acquisition decisions during execution, management, and oversight.
NDRI proposed some example topics where expanded analysis could potentially improve acquisition outcomes:
Recent highly publicized advances in commercial data analytics — including those involving artificial intelligence, machine learning, and big data — make it tempting to consider applications of these techniques to acquisition program management. But for a variety of reasons, DoD acquisition programs are not easily amenable to such applications. For example, DoD programs tend to fail for different reasons, and their numbers are low compared with the huge "training" data sets needed for predictive analytics. In addition, commercial successes using data analytics tend to emanate from highly planned efforts on the part of leadership (that is, top down).
Many of the individual acquisition functional domains have developed their own data management strategies. However, an overarching data analytics strategy is needed that provides key strategic questions and identifies the data needed to address those questions.
By leveraging private-sector best practices, the DoD has made progress in maturing data collection, access, and analysis in existing systems, although further progress has been hampered by concerns about data sharing. The importance of data governance in such areas as standardizing data definitions has been recognized. The DoD's program information managers recognize the importance of developing use cases to illustrate the need for data collection and analysis.
A persistent barrier to improving acquisition analytics uniformly and sharing data across the various functional communities is the stovepiping of acquisition data management.
Concerns about cybersecurity limit the expanded use of commercial software that would increase analytic capabilities. One possible solution is increased testing of commercial software and disseminating lists of safe analytic tools. Alternatively, the use of virtual computing environments can be used to run commercial software in isolation from DoD networks. Virtual environments solve the problem of isolating security concerns, but they impede data and information flowing in and out of the virtual environments.
Security concerns, as well as concerns about excessive oversight and distractions, have limited access to and sharing of data — not only with contractors who conduct data analytics for DoD acquisition domains but even across programs within the DoD and between the DoD and Congress. Although some recognize the need for data sharing, statutory authorities may be needed to establish and enforce sharing.
Data accessibility can be increased through several mechanisms. For example, Congress could grant permanent access to analysts in FFRDCs. However, other nongovernment analysts need access to particular data sources. An alternate idea is to develop DoD-wide data access categories, in which analysts would be granted blanket access by appropriate government officials.
Decisionmakers may benefit from ensuring that they have the incentives and authorities needed to appropriately balance insights from data analytics against other strategic considerations (e.g., related to policies, strategies, budgets, missions, urgency, and threats). Also, providing rising decisionmakers with the training and tools to understand how to interpret, weigh the strengths and limitations of, and apply relevant data to decisions could help strengthen the benefits of data analytics for decisionmaking.
The DoD's chart of accounts for research, development, test, and evaluation does not specifically track R&D for acquisition data analytics. NDRI analyzed the DoD FY 2019 budget request for indications of program elements that involved data analytics for acquisition. NDRI estimated that, across 31 program elements, approximately $200 million was requested based on analysis of the extent of data analytics in these program elements.
As for information technology systems related to acquisition, about $520 million was requested in FY 2019, an increase of $207 million from FY 2017.
Four topics related to acquisition data analytics were also identified in the January 2019 Small-Business Innovation Research (SBIR) and Small-Business Technology Transfer (STTR) solicitations. NDRI also found anecdotal evidence of exploratory research on acquisition analytics applications across the DoD.
These investments do not include R&D for military operations or other areas outside acquisition (e.g., budgeting, requirements, or intelligence).
NDRI studied the findings of consulting companies that assess, survey, and review the field for lessons learned and noted a fairly consistent set of common practices, including the following:
Information managers seek use cases to identify what data are needed and for what purposes. Designating authoritative data sources and sharing data across acquisition systems are becoming more common. The use of common program management software suites that can automatically share project or program data could be expanded.
Although the DoD has made progress in opening its data acquisition systems and sharing data, challenges to sharing remain. The most difficult problem is a culture that resists sharing. This resistance stems from a number of concerns, including security (both from elevated classification because of data aggregation and from unauthorized release of sensitive information), trust in how data are used, and appropriate data labeling. The DoD could encourage data sharing by emphasizing that these data are DoD enterprise assets, developing approaches to resolve security and sensitivity issues, and ensuring that oversight staff will not use data to micromanage programs.
Some anonymized personnel data (including acquisition workforce data) — which would otherwise be sensitive, personally identifiable information (PII) with legal releasability restrictions — are being made available through the Defense Manpower Data Center and the Office of Personnel Management.
Practical reasons explain why anonymization has not been widespread. Anonymization is not always reliable: Advances in analytic tools can sometimes identify data. Also, much of the metadata that would be removed in anonymization are important for analyzing potential causes of identified trends. In addition, DoD data generally lack data-sensitivity metadata at the data-element level, making it hard to select which data cannot be shared and why. Furthermore, government procedures for categorizing and handling sensitive data are complicated, slow, and not well understood by staff; incentives drive conservatism to block sharing (e.g., what exactly can and cannot be asserted as proprietary information by a contractor, how can markings be changed, and what are the personal risks involved?). Finally, other data are available without being anonymized. These include some program and budget data that are publicly released.
NDRI reviewed the curricula at four defense institutions: DAU, the Naval Postgraduate School, the Air Force Institute of Technology, and the National Defense University. Three of the four schools (DAU, the Air Force Institute of Technology, and the Naval Postgraduate School) offer a broad array of acquisition courses, ranging in depth and applicability from courses in acquisition theory and processes to hands-on applied data analytics courses, such as cost analysis, which represent the majority of the courses offered. These universities also offer courses in general purpose data analytics. The National Defense University focuses primarily on defense strategy, not acquisition.
DAU also has official partnerships with a number of civilian-sector universities and private-sector companies to offer classes to the DoD workforce, such as more-advanced coursework in data analytics. For example, partnerships with four universities in the District of Columbia area, Stanford University, the University of Michigan, and the Georgia Institute of Technology offer a wide selection of courses related to data analytics for acquisition, ranging from applied training to courses in policy. Private-sector partnerships include Google and IBM.
These applied and general-purpose courses should increase the ability of the acquisition workforce to conduct simple analysis while becoming smart consumers of analysis conducted by specialists. Still, it is unreasonable to expect or want most acquisition personnel to become experts in data analytics.
Personnel with expertise in both data analytics and the application domain are a rarity — not only in the DoD but in the private sector as well. Thus, a more achievable goal may be to develop an acquisition workforce that possesses the necessary range of skills and expertise to conduct, understand, and apply the findings of acquisition data analysis while growing a cadre of application specialists.
According to the findings of the report, DoD leaders need to identify what they want data analytics to accomplish, which will help define what specific acquisition data and analytic capabilities they need and what Congress and others can do to help. In the spirit of helping to address those questions, NDRI offers several suggested opportunities and next steps, categorized by stakeholder group.
Congress can take the following steps to help the DoD move acquisition data analytics forward.
Finally, NDRI recommends that DoD data analysts consider developing or expanding five areas of data analysis:
Although some of these recommended efforts are well underway, some will require further research to develop optional implementation approaches.