Assessing the Value of Structured Analytic Techniques in the U.S. Intelligence Community

by Stephen Artner, Richard S. Girven, James B. Bruce

Download Free Electronic Document

FormatFile SizeNotes
PDF file 0.2 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. How can the IC assess the overall effectiveness of structured analytic techniques?
  2. What approaches can best measure how frequently intelligence analysis employs structured techniques and which specific techniques are most effective in addressing the various types of intelligence questions?
  3. What are the most promising methods for compiling lessons learned regarding best practices and pitfalls in the use of structured analytic techniques?

Structured analytic techniques (SATs) are a key part of the rigorous analytic tradecraft the Intelligence Community (IC) has pursued in recent years, but so far these techniques have received little systematic evaluation. This report argues that the assessment of SATs is essential, albeit difficult; suggests specific questions that should be part of that assessment; and proposes several methods for ascertaining the practical value of SATs. The report also offers the results of a pilot study that explored conducting such an evaluation via structured interviews with analytic practitioners and a qualitative assessment of a body of IC products to examine the incidence and utility of SATs. These preliminary efforts do not offer definitive conclusions about the value of SATs but illustrate how the IC might evaluate them more systematically.

Key Findings

The Intelligence Community Is Strongly Emphasizing the Use of Structured Analytic Techniques to Promote Rigorous Analysis But Has Made Little Effort to Assess Whether They Are Measurably Improving the Quality of Analysis

  • One primarily qualitative method to evaluate these techniques would be periodic in-depth reviews of intelligence production on a variety of topics to ascertain how frequently structured techniques are used and which ones are most effective.
  • In addition, quantitative research could examine the extent to which the use or nonuse of structured techniques correlates with the quality of analytic output as measured by product evaluation staffs, while controlled experiments could test these methods' contribution to intelligence analysis.

RAND's Preliminary Review of a Limited Sample of Intelligence Publications Finds That the Minority of Those Employing Structured Techniques Addressed a Broader Range of Potential Outcomes and Implications Than Did Other Analyses

  • In some cases, however, key assumptions and the logic behind specific techniques were not transparent.
  • More comprehensive reviews, along with interviews of analysts, managers, and consumers, could determine how effectively agencies are employing structured techniques and provide lessons learned on best practices and pitfalls of their use in intelligence analysis.

This research was conducted within the Intelligence Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.