SimCoach Evaluation

A Virtual Human Intervention to Encourage Service-Member Help-Seeking for Posttraumatic Stress Disorder and Depression

by Daniella Meeker, Jennifer L. Cerully, Megan Johnson, Neema Iyer, Jeremy Kurz, Deborah M. Scharf

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 3.7 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Purchase

Purchase Print Copy

 FormatList Price Price
Add to Cart Paperback144 pages $49.50 $39.60 20% Web Discount

Research Questions

  1. Did the SimCoach development process align with best practices in software engineering and clinical intervention development?
  2. Is an online, virtual-reality intervention an appealing and efficacious tool for encouraging service members to seek help for psychological health concerns?
  3. Is the beta version of SimCoach itself acceptable and effective for promoting help-seeking?

This report describes RAND Corporation researchers' assessment of SimCoach, a computer program featuring a virtual human that speaks and gestures in a video game–like interface, designed to encourage service members, especially those with signs or symptoms of posttraumatic stress disorder (PTSD) or depression, to seek help to improve their psychological health. The assessment included a formative component assessing SimCoach's design, development, and implementation approaches and a summative component assessing outcomes among participants in a user experience survey and a randomized controlled trial (RCT). Results of the formative evaluation identified both strengths and opportunities for improvement. For example, although SimCoach development processes were well-aligned with best practices for software engineering, SimCoach content development and evaluation processes could have been more tightly coupled to best practices in psychological health. The summative evaluation RCT did not show any SimCoach-related benefit in intent to seek help compared with that of control users not exposed to any intervention. However, secondary outcomes indicated that SimCoach users had satisfying experiences without distress. If SimCoach development is continued, greater attention to clinical processes and outcomes is needed so that the program can have its intended impact on help-seeking for PTSD and depression.

Key Findings

The Formative and Summative Evaluations Yielded Similar Findings

  • The main finding from the formative evaluation is that the development of SimCoach had a greater focus on user experiences than on the outcome of interest for our study — the efficacy of SimCoach in improving users' intentions to seek help.
  • The main finding from the summative evaluation is that participants who used SimCoach did not report greater help-seeking intentions than participants receiving no intervention.

Recommendations

  • These recommendations are for both the SimCoach developers and the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE), which funded SimCoach development.
  • SimCoach developers should implement best practices for the development of help-seeking interventions; consider new approaches to SimCoach marketing to promote the novelty of the intervention; use validated screening instruments when possible to ensure that these instruments have sufficient reliability, sensitivity, and specificity; consider using an outcome-oriented, iterative development process during subsequent improvements to the SimCoach intervention; continue to design new dialogue and content to meet SimCoach goals of reaching a target audience of service members, veterans, and family members; if future versions are found to be effective, develop versions of SimCoach that are compatible with mobile devices and web browsers; and consider using SimCoach in other cases in which potentially sensitive questionnaires and information may be delivered.
  • When considering involvement with SimCoach and similar programs, DCoE should consider changing funding models to motivate best practices in intervention development (e.g., requiring submission of pilot data prior to funding larger technology-development projects or requiring the use of validated instruments); support pilot evaluations and dissemination approaches in different contexts; consider investing in strategies to guide the development of technology-based clinical interventions; and play an active role in design and monitoring of outcome-oriented progress metrics for technology-development projects.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Formative Evaluation

  • Chapter Three

    Summative Evaluation

  • Chapter Four

    Summary and Recommendations

  • Appendix A

    Revisions to SimCoach Beta Recommendations

  • Appendix B

    Additional Distress Signal Phrases

  • Appendix C

    SimCoach Beta Participant Comments

  • Appendix D

    SimCoach Proposal

  • Appendix E

    SimCoach Onsite Evaluation: Preliminary Results

This research was sponsored by DCoE and conducted within the Forces and Resources Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.