Toward Evidence-Based Measures of Implementation

Examining the Relationship Between Implementation Outcomes and Client Outcomes

Published in: Journal of Substance Abuse Treatment, v. 67, Aug. 2016, p. 15-21

Posted on RAND.org on May 25, 2016

by Bryan R. Garner, Sarah B. Hunter, Beth Ann Griffin, Susan H Godley

Read More

Access further information on this document at Journal of Substance Abuse Treatment

This article was published outside of RAND. The full text of the article can be found at the link above.

Research Question

  1. What evidence-based measures of implementation predict better client outcomes in the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based practice for adolescent substance use treatment?

Background

Developing consistent, valid, and efficient implementation outcome measures is necessary to advance implementation science. However, development of such measures has been limited to date, especially for validating the extent to which such measures are associated with important improvements in client outcomes. This study seeks to address this gap by developing one or more evidence-based measures of implementation (EBMIs; i.e., implementation outcome measure that is predictive of improvements in key client outcomes) for the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based practice (EBP) for adolescent substance use.

Methods

Data for the current study were collected as part of a large-scale federally funded EBP dissemination and implementation initiative. The multilevel dataset included 65 substance use treatment organizations, 308 clinicians, and 5873 adolescent clients. Adjusted multilevel regression analyses were used to examine the extent to which client-level outcome measures assessed at 6-month follow-up (i.e., substance use, emotional problems) could be predicted by four implementation outcomes: two measures of fidelity (i.e., session exposure, procedure exposure) and two measures of penetration (i.e., absolute client penetration, absolute staff penetration).

Results

Adjusting for client substance use at intake, as well as several client characteristics (e.g., age, race, criminal justice involvement), client substance use at follow-up was significantly lower for treatment organizations that had higher procedure exposure (B = − 1.227, standard error [SE] = 0.583, 95% confidence interval = − 2.370, 0.252; p < .05). None of the other three implementation outcome measures were found to predict improvements in client outcomes.

Conclusions

The current study provides support for procedure exposure as an organizational-level EBMI for A-CRA. Thus, future efforts focused on implementing A-CRA could be improved by measuring and monitoring the extent to which A-CRA procedures are being delivered to clients. Additionally, given the dearth of studies that have examined the relationship between organizational-level measures of implementation and client outcomes, this article provides a prototype for future research to identify EBMIs for other behavioral treatments.

Key Findings

  • The number of discrete A-CRA procedures implemented in a treatment program predicted client outcomes.
  • Other measures of implementation—e.g., the number of A-CRA sessions implemented, the number of clients, or the number of staff trained in A-CRA treatment did not predict improvements in client outcomes.

Recommendation

Future efforts to implement A-CRA should measure and monitor the extent to which A-CRA procedures are being delivered to clients

This report is part of the RAND Corporation external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.