This report argues for more-systematic assessment of the structured analytic techniques increasingly employed by the Intelligence Community. It also presents a pilot study illustrating how such an assessment might be conducted and suggests additional approaches for evaluating the contribution of structured techniques to intelligence analysis.
Assessing the Value of Structured Analytic Techniques in the U.S. Intelligence Community
Download Free Electronic Document
| Format | File Size | Notes |
|---|---|---|
| PDF file | 0.2 MB | Use Adobe Acrobat Reader version 10 or higher for the best experience. |
Research Questions
- How can the IC assess the overall effectiveness of structured analytic techniques?
- What approaches can best measure how frequently intelligence analysis employs structured techniques and which specific techniques are most effective in addressing the various types of intelligence questions?
- What are the most promising methods for compiling lessons learned regarding best practices and pitfalls in the use of structured analytic techniques?
Structured analytic techniques (SATs) are a key part of the rigorous analytic tradecraft the Intelligence Community (IC) has pursued in recent years, but so far these techniques have received little systematic evaluation. This report argues that the assessment of SATs is essential, albeit difficult; suggests specific questions that should be part of that assessment; and proposes several methods for ascertaining the practical value of SATs. The report also offers the results of a pilot study that explored conducting such an evaluation via structured interviews with analytic practitioners and a qualitative assessment of a body of IC products to examine the incidence and utility of SATs. These preliminary efforts do not offer definitive conclusions about the value of SATs but illustrate how the IC might evaluate them more systematically.
Key Findings
The Intelligence Community Is Strongly Emphasizing the Use of Structured Analytic Techniques to Promote Rigorous Analysis But Has Made Little Effort to Assess Whether They Are Measurably Improving the Quality of Analysis
- One primarily qualitative method to evaluate these techniques would be periodic in-depth reviews of intelligence production on a variety of topics to ascertain how frequently structured techniques are used and which ones are most effective.
- In addition, quantitative research could examine the extent to which the use or nonuse of structured techniques correlates with the quality of analytic output as measured by product evaluation staffs, while controlled experiments could test these methods' contribution to intelligence analysis.
RAND's Preliminary Review of a Limited Sample of Intelligence Publications Finds That the Minority of Those Employing Structured Techniques Addressed a Broader Range of Potential Outcomes and Implications Than Did Other Analyses
- In some cases, however, key assumptions and the logic behind specific techniques were not transparent.
- More comprehensive reviews, along with interviews of analysts, managers, and consumers, could determine how effectively agencies are employing structured techniques and provide lessons learned on best practices and pitfalls of their use in intelligence analysis.
Research conducted by
This research was conducted within the Intelligence Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.