Toward Evidence-Based Measures of Implementation

Examining the Relationship Between Implementation Outcomes and Client Outcomes

Published in: Journal of Substance Abuse Treatment, v. 67, Aug. 2016, p. 15-21

Posted on on May 25, 2016

by Bryan R. Garner, Sarah B. Hunter, Beth Ann Griffin, Susan H. Godley

Read More

Access further information on this document at Journal of Substance Abuse Treatment

This article was published outside of RAND. The full text of the article can be found at the link above.

Research Question

  1. What evidence-based measures of implementation predict better client outcomes in the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based practice for adolescent substance use treatment?


Developing consistent, valid, and efficient implementation outcome measures is necessary to advance implementation science. However, development of such measures has been limited to date, especially for validating the extent to which such measures are associated with important improvements in client outcomes. This study seeks to address this gap by developing one or more evidence-based measures of implementation (EBMIs; i.e., implementation outcome measure that is predictive of improvements in key client outcomes) for the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based practice (EBP) for adolescent substance use.


Data for the current study were collected as part of a large-scale federally funded EBP dissemination and implementation initiative. The multilevel dataset included 65 substance use treatment organizations, 308 clinicians, and 5873 adolescent clients. Adjusted multilevel regression analyses were used to examine the extent to which client-level outcome measures assessed at 6-month follow-up (i.e., substance use, emotional problems) could be predicted by four implementation outcomes: two measures of fidelity (i.e., session exposure, procedure exposure) and two measures of penetration (i.e., absolute client penetration, absolute staff penetration).


Adjusting for client substance use at intake, as well as several client characteristics (e.g., age, race, criminal justice involvement), client substance use at follow-up was significantly lower for treatment organizations that had higher procedure exposure (B = − 1.227, standard error [SE] = 0.583, 95% confidence interval = − 2.370, 0.252; p < .05). None of the other three implementation outcome measures were found to predict improvements in client outcomes.


The current study provides support for procedure exposure as an organizational-level EBMI for A-CRA. Thus, future efforts focused on implementing A-CRA could be improved by measuring and monitoring the extent to which A-CRA procedures are being delivered to clients. Additionally, given the dearth of studies that have examined the relationship between organizational-level measures of implementation and client outcomes, this article provides a prototype for future research to identify EBMIs for other behavioral treatments.

Key Findings

  • The number of discrete A-CRA procedures implemented in a treatment program predicted client outcomes.
  • Other measures of implementation—e.g., the number of A-CRA sessions implemented, the number of clients, or the number of staff trained in A-CRA treatment did not predict improvements in client outcomes.


Future efforts to implement A-CRA should measure and monitor the extent to which A-CRA procedures are being delivered to clients

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behavior. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.