How to evaluate counterterrorism policies and interventions

Dutch police on security patrol

George Pachantouris/Getty Images

Initiatives to counter terrorism and violent extremism are rarely evaluated and are often designed on the basis of untested assumptions. Researchers developed an inventory of 48 published and unpublished evaluations conducted between 2013 and 2017, as well as an analytical framework for assessing these evaluations.

Based on their assessment, using the framework they developed, the research team made several recommendations, including developing new evaluation designs, frameworks and approaches for conducting evaluations, and encouraging the adoption of (quasi-)experimental designs, where appropriate.

Background

Recent years have seen an increase in terrorist and violent extremist incidents occurring across Europe. In response to this, the volume of policies and programmes adopted for counterterrorism (CT) and preventing and countering violent extremism (PCVE) has rapidly increased.

However, owing to the political imperative and speed with which these policies and programmes are implemented, such initiatives are rarely evaluated and are often designed on the basis of untested assumptions.

Goals

The Research and Documentation Centre (Wetenschappelijk Onderzoek- en Documentatiecentrum, WODC) of the Dutch Ministry of Justice and Security commissioned RAND Europe to conduct a study aimed at investigating how evaluations of CT and PCVE policies in the Netherlands and abroad have been designed and conducted over the past five years, and what practical lessons can be drawn regarding such evaluations.

Methodology

The study employed a mixed methodology approach comprising three interconnected tasks:

  • Developing an inventory of published and unpublished CT and PCVE evaluations conducted in the past five years (2013-2017)
  • Developing an analytical framework for assessing CT and PCVE evaluations
  • Analysing the CT and PCVE evaluations gathered through the lenses of the study analytical framework

Findings

The study produced an inventory comprising 48 CT and PCVE evaluations manuscripts (38 in English, six in Dutch and four in German). This suggests that a growing volume of CT and PCVE evaluations have been undertaken in recent years, and that the majority of these rely on primary data from multiple sources, perspectives and methods.

Nonetheless, there appear to be limits to the extent to which evaluation practice has advanced and grown evenly across all areas of CT and PCVE work as significant gaps and shortcomings continue to mar a number of evaluations (e.g., evaluations characterised by designs that undermine their ability to draw robust conclusions about an initiative's impact).

Not all the issues and challenges are exclusive to the fields of CT and PCVE however. Lessons and reflections identified from evaluations reviewed pertain to:

  1. inherent complexities of the fields of CT and PCVE;
  2. challenges associated with measuring real-world phenomena;
  3. challenges associated with existing evaluation designs;
  4. practical difficulties of conducting evaluations; and
  5. drawbacks and benefits of specific evaluation methods.

Recommendations

  • Continue to invest in the evaluation of planned and existing initiatives in the CT and PCVE policy areas, setting minimum quality and robustness requirements for future evaluations.
  • Develop new evaluation designs, frameworks and approaches for conducting evaluation in the CT and PCVE policy areas, encouraging the adoption wherever possible of (quasi-)experimental designs.
  • Encourage further research on the dynamics, drivers and factors governing the phenomena of radicalisation, violent extremism and terrorism.
  • Conduct regular mapping and stocktaking exercises akin to the present study, providing resources and means required to gain access to CT and PCVE initiatives' evaluators and beneficiaries.