Aug 21, 2014
Photo by Cpl. Artur Shvartsberg/U.S. Marine Corps
Locally focused stability operations are designed to help host nation citizens better protect and govern themselves. How can practitioners tell if such efforts are working?
Locally focused stability operations (LFSO) are a "bottom-up" approach to fostering security, development, and governance, working with and through communities experiencing conflict. LFSO are often facilitated by U.S. military advisors but ideally are a whole-of-government effort led by the host nation (HN). In recent years, U.S. and NATO forces practiced LFSO to help villages in Afghanistan create their own defensively oriented local police units, bring in appropriate development projects, and create links with their district governments. This approach continues to foster security in those areas.
The complexity of the operational environments in which LFSO take place makes it difficult for LFSO practitioners to determine whether their efforts are successful. Every LFSO is distinctive, and, as such, there is no established doctrine for measuring LFSO progress and outcomes. Many practitioners have developed their own set of assessment tools, with varying degrees of success. Effective assessment is important, however, as it can help answer such key questions as:
In addition, assessments can do more than support a specific LFSO effort. If done right, they can suggest best practices to inform evolving doctrine for stability operations.
To help with the planning and conduct of LFSO assessment, RAND researchers (1) identified challenges related to LFSO assessment, (2) surveyed relevant practices, and (3) created concrete guidance for assessment planning, based on reviews of available tools, findings of prior research, and input from assessment experts and LFSO practitioners. This document compiles best practices for designing LFSO assessments and offers a notional scenario that illustrates a five-step approach to LFSO assessment.
Assessments should reflect how and why the commander believes the tasks that have been laid out will result in the desired end-state. This involves documenting the Theory of Change. As illustrated below, the Theory of Change is the underlying structure that connects activities, desired outcomes (effects), and superordinate objectives through a "chain of consequences" and illustrates how outputs at a given stage may become inputs at the next stage. A clearly articulated Theory of Change will enable the assessment team to identify appropriate inputs, outputs, and outcomes to measure.
Assessment process should start during campaign or mission planning and should be reviewed and assessed regularly to determine whether changes are needed.
Assessment teams should fully exploit the data and methods available, leveraging the strengths of a given source against the weaknesses of others to obtain valid estimates for the ground truth.
The study identified a number of challenges that commanders and assessment experts face when conducting LFSO assessments. The table below summarizes these challenges and offers solutions to address them.
|LSFO Assessment Challenges||Recommended Solutions|
|Assessing the impact of stability operations in a complex environment is not easy.||Identify the challenges, take a deep breath, and forge ahead.|
Doctrine and training fail to adequately address the complexities of assessment and create appropriate skill sets.
Prioritize assessment-related doctrine/training.
Institutionalize the assessor role.
Assign individuals with the "right" personality traits.
Have subject-matter experts help fill in contextual gaps.
Allow analysts to deploy to gain operational grounding.
The United States, coalition partners, HNs, nongovernmental organizations, and other stakeholders may have competing visions of stability.
Establish an interagency/international working group to identify a set of variables to track across all lines of effort (security, governance, and development).
Develop an off-the-shelf assessment capability that uses a standard framework and is accepted by the broader stakeholder community.
|There is a strategic- vs. tactical-level challenge: Too much aggregation of data obfuscates nuance, too little can overwhelm consumers.||Present results in a way that efficiently and clearly summarizes but can support more detailed exploration of data should the need arise.|
|Assessments sometimes rely on invalid or untested assumptions about causes and effects.||
Avoid drawing hasty conclusions by identifying/documenting and testing/validating assumptions.
Adjust the Theory of Change (see next page) accordingly.
|Bias, conflicts of interest, and other external factors can create perverse incentives.||Employ diverse sources and methods to validate data, using observable indicators, devil's advocacy, ratios, and other tools.|
|Redundant reporting requirements and knowledge-management challenges impede the assessment process.||
Ask for data less frequently but require more in-depth responses or ask for data more often but use less onerous questions.
Provide direct benefits (e.g., tailored products) to those who provide data to validate and motivate.
|Maintaining the continuity of the assessment process can be difficult across deployment cycles.||Plan for personnel turnover (training, documentation, knowledge management).|
|Assessment planning often ignores HN perspectives.||
Invite HN participation in assessment planning and execution.
Carefully consider hidden agendas.
Here we show how commanders and an LFSO assessment team might put our recommendations into action through an LFSO scenario based in a notional West African country in 2015. The operation order below describes the operational environment and presents the concept of operations. The Theory of Change, and how assessment should proceed in this particular case, is presented further below.
The commander's LFSO Theory of Change should guide all aspects of LFSO assessment, including which metrics to use. The metrics are organized by line of effort (security, development, governance, stability), metric type (quantitative, narrative, or mixed data), collection frequency, and the parties charged with information collection (the assessment team, LFSO teams in the villages, HN staff, or others). The table below presents a sample of local security-related metrics that the notional assessment team in this scenario might consider, based on the Theory of Change outlined on the next page. Of course, there are many other kinds of metrics available for LFSO assessment. A more comprehensive list, along with a fuller description of the scenario, is provided in the associated RAND report.
|Number of members of guard force that meet proficiency and equipping standards||Quantitative||Biweekly||LFSO team|
|Number of guard force patrols per week||Contributes to feeling of security by villagers, deters insurgents, helps gather intelligence, and demonstrates guard force capability and capacity||Quantitative||Biweekly||LFSO team|
|Level of insurgent attacks in area of operations||Mixed||Biweekly||LFSO team|
|Insurgent (INS) influence/propaganda/ intimidation efforts in area of operations||Mixed||Biweekly||LFSO team, intelligence team|
|Number of calls to tip line from area of operations||Indicates level of intimidation, and pro-HN-government attitude of villagers||Quantitative||Biweekly||LFSO team|
|Villager perceptions of security||Narrative||Quarterly (survey team), biweekly (LFSO team)||Survey team, LFSO team|
|Children moving around unaccompanied||Validates survey data about villager perceptions of security||Narrative||Biweekly||LFSO team|
Identify the challenges specific to the scenario.
Specific challenges to the reliable and valid assessment of the LFSO need to be identified and documented at the beginning of the planning process. Assessment in this scenario is difficult, mainly because of the complex mission design and the number of organizations involved in the effort. The assessment team must also identify and mitigate the potential for biased data stemming from organizational self-interest.
Establish the Theory of Change.
The Theory of Change behind the operation will inform the kinds of metrics to use and the data to be collected, and focuses the assessment by clarifying its purpose, identifying relevant influences to consider, and helping the assessment team choose appropriate metrics for measuring impact. For example, an LFSO Theory of Change would assert that sustainable security in a village requires a whole-government approach integrating increased security, economic development, and improved governance. (See figure below.)
Determine metrics and how/when to collect them.
When determining appropriate metrics for measuring impact, the following should be specified: Metric type, such as Quantitative data — numbers, ratios (fractions); ordinal metrics (discrete, ranked categories, such as "high," medium," and "low"), nominal metrics (discrete, non-ranked categories, such as "nationality"); Qualitative data — narrative data, descriptive text, etc.; and Mixed data — text containing quantitative information; Collection frequency, such as biweekly, monthly, quarterly, etc.; and Responsibility, such as who is responsible for collecting the information, other actions, etc. See the previous page for a sample of metrics that might be used in this scenario.
Set up processes for data analysis (including aggregation) and communication of results.
Assessment planning must include determining how best to communicate results to decisionmakers. Practitioners should remember that too much aggregation of data obfuscates nuance, but too little aggregation can overwhelm analysis. Results should be presented in a way that efficiently and clearly summarizes results but can support a more detailed exploration of data should the need arise.
Brief leadership and stakeholders on the assessment plan.
Once the assessment team has gone through the planning process outlined above, the results should be discussed with the leaders involved in the LFSO effort, in order to obtain additional guidance and, after potentially revising the assessment plan to address any concerns, approval and support from the organizations involved.
LFSO are a powerful way for the United States and allied forces to help destabilized areas create safety, foster economic growth, and establish accountable governance. However, LFSO practitioners must make sure that their hard work is actually helping to achieve those strategic goals. LFSO assessment thus is a vital activity, that, if done properly, gives LFSO practitioners and their commanders the necessary perspective of their actions. The following principles should guide LFSO assessment efforts: