- Did the SimCoach development process align with best practices in software engineering and clinical intervention development?
- Is an online, virtual-reality intervention an appealing and efficacious tool for encouraging service members to seek help for psychological health concerns?
- Is the beta version of SimCoach itself acceptable and effective for promoting help-seeking?
This report describes RAND Corporation researchers' assessment of SimCoach, a computer program featuring a virtual human that speaks and gestures in a video game–like interface, designed to encourage service members, especially those with signs or symptoms of posttraumatic stress disorder (PTSD) or depression, to seek help to improve their psychological health. The assessment included a formative component assessing SimCoach's design, development, and implementation approaches and a summative component assessing outcomes among participants in a user experience survey and a randomized controlled trial (RCT). Results of the formative evaluation identified both strengths and opportunities for improvement. For example, although SimCoach development processes were well-aligned with best practices for software engineering, SimCoach content development and evaluation processes could have been more tightly coupled to best practices in psychological health. The summative evaluation RCT did not show any SimCoach-related benefit in intent to seek help compared with that of control users not exposed to any intervention. However, secondary outcomes indicated that SimCoach users had satisfying experiences without distress. If SimCoach development is continued, greater attention to clinical processes and outcomes is needed so that the program can have its intended impact on help-seeking for PTSD and depression.
The Formative and Summative Evaluations Yielded Similar Findings
- The main finding from the formative evaluation is that the development of SimCoach had a greater focus on user experiences than on the outcome of interest for our study — the efficacy of SimCoach in improving users' intentions to seek help.
- The main finding from the summative evaluation is that participants who used SimCoach did not report greater help-seeking intentions than participants receiving no intervention.
- These recommendations are for both the SimCoach developers and the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE), which funded SimCoach development.
- SimCoach developers should implement best practices for the development of help-seeking interventions; consider new approaches to SimCoach marketing to promote the novelty of the intervention; use validated screening instruments when possible to ensure that these instruments have sufficient reliability, sensitivity, and specificity; consider using an outcome-oriented, iterative development process during subsequent improvements to the SimCoach intervention; continue to design new dialogue and content to meet SimCoach goals of reaching a target audience of service members, veterans, and family members; if future versions are found to be effective, develop versions of SimCoach that are compatible with mobile devices and web browsers; and consider using SimCoach in other cases in which potentially sensitive questionnaires and information may be delivered.
- When considering involvement with SimCoach and similar programs, DCoE should consider changing funding models to motivate best practices in intervention development (e.g., requiring submission of pilot data prior to funding larger technology-development projects or requiring the use of validated instruments); support pilot evaluations and dissemination approaches in different contexts; consider investing in strategies to guide the development of technology-based clinical interventions; and play an active role in design and monitoring of outcome-oriented progress metrics for technology-development projects.
Table of Contents
Summary and Recommendations
Revisions to SimCoach Beta Recommendations
Additional Distress Signal Phrases
SimCoach Beta Participant Comments
SimCoach Onsite Evaluation: Preliminary Results