Best Practices for Assessing Locally Focused Stability Operations

Photo by Cpl. Artur Shvartsberg/U.S. Marine Corps

Locally focused stability operations are designed to help host nation citizens better protect and govern themselves. How can practitioners tell if such efforts are working?

Locally focused stability operations (LFSO) are a "bottom-up" approach to fostering security, development, and governance, working with and through communities experiencing conflict. LFSO are often facilitated by U.S. military advisors but ideally are a whole-of-government effort led by the host nation (HN). In recent years, U.S. and NATO forces practiced LFSO to help villages in Afghanistan create their own defensively oriented local police units, bring in appropriate development projects, and create links with their district governments. This approach continues to foster security in those areas.

Photo by Spc. Tia Sokimson/U.S. Army

The complexity of the operational environments in which LFSO take place makes it difficult for LFSO practitioners to determine whether their efforts are successful. Every LFSO is distinctive, and, as such, there is no established doctrine for measuring LFSO progress and outcomes. Many practitioners have developed their own set of assessment tools, with varying degrees of success. Effective assessment is important, however, as it can help answer such key questions as:

• Are the mission objectives being met?
• Do LFSO personnel have what they need to achieve mission objectives?
• Do new developments require adjustments in objectives or methods?

In addition, assessments can do more than support a specific LFSO effort. If done right, they can suggest best practices to inform evolving doctrine for stability operations.

To help with the planning and conduct of LFSO assessment, RAND researchers (1) identified challenges related to LFSO assessment, (2) surveyed relevant practices, and (3) created concrete guidance for assessment planning, based on reviews of available tools, findings of prior research, and input from assessment experts and LFSO practitioners. This document compiles best practices for designing LFSO assessments and offers a notional scenario that illustrates a five-step approach to LFSO assessment.

Best Practices for LFSO Assessment

Assessments should be based on a clear "Theory of Change."

Assessments should reflect how and why the commander believes the tasks that have been laid out will result in the desired end-state. This involves documenting the Theory of Change. As illustrated below, the Theory of Change is the underlying structure that connects activities, desired outcomes (effects), and superordinate objectives through a "chain of consequences" and illustrates how outputs at a given stage may become inputs at the next stage. A clearly articulated Theory of Change will enable the assessment team to identify appropriate inputs, outputs, and outcomes to measure.

Assessments should be commander-centric.

Assessment process should start during campaign or mission planning and should be reviewed and assessed regularly to determine whether changes are needed.

Assessments should "triangulate the truth."

Assessment teams should fully exploit the data and methods available, leveraging the strengths of a given source against the weaknesses of others to obtain valid estimates for the ground truth.

Challenges and Solutions for LFSO Assessment

The study identified a number of challenges that commanders and assessment experts face when conducting LFSO assessments. The table below summarizes these challenges and offers solutions to address them.

LFSO Assessment Planning Example

Here we show how commanders and an LFSO assessment team might put our recommendations into action through an LFSO scenario based in a notional West African country in 2015. The operation order below describes the operational environment and presents the concept of operations. The Theory of Change, and how assessment should proceed in this particular case, is presented further below.

Assessment Metric Choice Should Be Guided by LFSO Theory of Change

The commander's LFSO Theory of Change should guide all aspects of LFSO assessment, including which metrics to use. The metrics are organized by line of effort (security, development, governance, stability), metric type (quantitative, narrative, or mixed data), collection frequency, and the parties charged with information collection (the assessment team, LFSO teams in the villages, HN staff, or others). The table below presents a sample of local security-related metrics that the notional assessment team in this scenario might consider, based on the Theory of Change outlined on the next page. Of course, there are many other kinds of metrics available for LFSO assessment. A more comprehensive list, along with a fuller description of the scenario, is provided in the associated RAND report.

Security-Related Metrics

Five Steps to LFSO Assessment

1. Identify the challenges specific to the scenario.

   Specific challenges to the reliable and valid assessment of the LFSO need to be identified and documented at the beginning of the planning process. Assessment in this scenario is difficult, mainly because of the complex mission design and the number of organizations involved in the effort. The assessment team must also identify and mitigate the potential for biased data stemming from organizational self-interest.

2. Establish the Theory of Change.

   The Theory of Change behind the operation will inform the kinds of metrics to use and the data to be collected, and focuses the assessment by clarifying its purpose, identifying relevant influences to consider, and helping the assessment team choose appropriate metrics for measuring impact. For example, an LFSO Theory of Change would assert that sustainable security in a village requires a whole-government approach integrating increased security, economic development, and improved governance. (See figure below.)

3. Determine metrics and how/when to collect them.

   When determining appropriate metrics for measuring impact, the following should be specified: Metric type, such as Quantitative data — numbers, ratios (fractions); ordinal metrics (discrete, ranked categories, such as "high," medium," and "low"), nominal metrics (discrete, non-ranked categories, such as "nationality"); Qualitative data — narrative data, descriptive text, etc.; and Mixed data — text containing quantitative information; Collection frequency, such as biweekly, monthly, quarterly, etc.; and Responsibility, such as who is responsible for collecting the information, other actions, etc. See the previous page for a sample of metrics that might be used in this scenario.

4. Set up processes for data analysis (including aggregation) and communication of results.

   Assessment planning must include determining how best to communicate results to decisionmakers. Practitioners should remember that too much aggregation of data obfuscates nuance, but too little aggregation can overwhelm analysis. Results should be presented in a way that efficiently and clearly summarizes results but can support a more detailed exploration of data should the need arise.

5. Brief leadership and stakeholders on the assessment plan.

   Once the assessment team has gone through the planning process outlined above, the results should be discussed with the leaders involved in the LFSO effort, in order to obtain additional guidance and, after potentially revising the assessment plan to address any concerns, approval and support from the organizations involved.