Cross-Agency Evaluation of DoD, VA, and HHS Mental Health Public Awareness Campaigns: Analysis of Campaign Scope, Content, and Reach
Jan 29, 2020
Photo by Anna-Victoria Crisostomo/U.S. Navy
When service members return from conflicts abroad, some bring with them depression, posttraumatic stress disorder, substance use disorders, and other mental health issues that can affect their quality of life, their ability to function at home and on the job, and their relationships with family and friends. Many do not realize that they need help, or they hesitate to seek treatment because they fear that it will have negative repercussions on their careers. Without care, these service members and veterans may face lifelong challenges.
In 2015, the Obama administration designated improving mental health outcomes for service members, veterans, and their families as a Cross-Agency Priority Goal (CAP-G). In response, the Executive Office of the President, U.S. Department of Defense (DoD), U.S. Department of Veterans Affairs (VA), and U.S. Department of Health and Human Services (HHS) commissioned an evaluation (funded by the DoD Psychological Health Center of Excellence) of several mental health public awareness campaigns to better understand the degree to which they help people overcome negative perceptions of mental health disorders and treatment-seeking and promote awareness of mental health resources.
With the goals of better understanding whether these campaigns are working and identifying ways to improve the federal approach to this issue, an interagency task force asked RAND to conduct an evaluation of four such efforts: the Real Warriors Campaign (DoD), Make the Connection (VA), outreach efforts of the Veterans Crisis Line (VA), and Recovery Month (HHS). This brief describes the results of the cross-agency evaluation, which examined the scope, content, and dissemination of the campaigns.
The RAND team's cross-agency evaluation sought to understand the campaign's collective scope, content, and dissemination by analyzing campaign processes, material, and social media data. The cross-agency evaluation was designed to address goals, messages, and processes shared across multiple agencies and campaigns, so the results presented here are not intended to serve as a comprehensive evaluation of each campaign, its effectiveness, or its cost-effectiveness.
The following sections summarize the RAND team's findings and recommendations in four areas: efficiency and mental health messaging, content of campaign materials, dissemination and reach, and inclusion of resources that directly connect individuals to care. At the end of each recommendation, we note in parentheses who should take the lead in addressing that issue—a federal partner group (such as an interagency task force or working group) or the campaigns themselves.
of content across campaigns aligns with shared goals
The four campaigns, which have distinct yet overlapping audiences, share targeted outcomes (to improve knowledge of mental health symptoms and create positive perceptions of individuals with mental health conditions and of mental health treatment). They offer a variety of messages—for those who are at risk of mental illness, considering treatment, in need of immediate support, or in recovery—to their common audiences and unique messages when appropriate.
A majority of campaign content (82 percent across all four) aligns with shared goals, and the individuals pictured in the materials reflect the target audiences of each campaign. The campaigns could coordinate more with each other and their shared partners, and could cross-reference each other more as resources.
Because campaigns were initiated at different points in time, no shared measures were being used across campaigns to facilitate tracking across overlapping target populations.
of the campaign websites did not specifically cite the sources used to generate content for their webpages
Most campaign content aligned with best practices; clearly communicated the messages; and used credible, positive messengers. All of the campaigns use testimonials from individuals about their own experiences with mental illness, substance use, and recovery, which can help reduce negative perceptions of people with mental illness and increase positive perceptions.
However, the campaigns' content could be improved in several areas. Half of the campaign websites did not cite the sources of the information used on webpages, and some campaign content did not align with a specific campaign outcome. Campaign sites housed some faulty links or outdated content that should be updated. Some experts who reviewed the campaigns thought that they could be clearer about how service members' anonymity or confidentiality is protected. In addition, campaigns that tried to serve multiple audiences often had content for secondary audiences (e.g., family members and health professionals) that the expert panel found limited and less developed than content focused on each campaign's main audience.
More than 4 million
visits were made to the campaign websites in 2015
Campaigns used multiple approaches to disseminate their materials and messages, including websites, public service announcements (PSAs), social media, television, radio, and in-person engagement. While the reach of all campaigns (except for Recovery Month) increased between 2012 and 2015, it was impossible to measure the degree to which campaigns reached their specific target audiences because campaigns were not collecting that data.
The analysis found that the campaigns reached as many as 5.6 million people in 2015 through Facebook, YouTube, and Twitter. VA's Make the Connection campaign had the greatest reach on these platforms, providing more than 90 percent of the campaigns' collective Facebook fans and YouTube views. Although campaigns used Twitter in a very limited way, they still managed to connect with users who have above-average influence in dissemination of tweets about mental health.
The campaigns' websites played host to more than 4 million visitor sessions. However, a significant share of visitors spent only a brief amount of time on the websites and had minimal engagement.
The campaigns aired more than 400,000 radio and television PSAs, resulting in more than 42.9 billion impressions (the number of audience members who might have been exposed to the PSA). Ninety percent of impressions came from DoD's Real Warriors Campaign. The campaigns also distributed more than 10 million campaign brochures, pieces of merchandise, and other items through in-person outreach at more than 250 events and by partnering with more than 700 organizations and agencies. The timing of outreach varied, with some campaigns highly active during certain time periods (particularly Recovery Month). It is unclear to what degree campaigns coordinated the timing of their activities.
of materials provided a direct connection to care
Although all campaigns offered general resources on mental health problems and treatment (e.g., locator tools that allow users to find care nearby, outreach phone lines that provide general resources), just 27 percent of campaign materials provided direct connections to care (defined as resources that connect to a crisis chat or phone line or connect directly to a specific medical center to make an appointment).
Providing direct connections to care may be a difficult task, but this approach is relevant to the campaigns' shared goal of encouraging target audiences to seek care if needed.
The cross-agency evaluation of the four DoD, VA, and HHS mental health public awareness campaigns detailed in this report relied on three assumptions: (1) To align with CAP-G, the evaluation should focus on service members, veterans, and their families; (2) to be efficient, the evaluation should focus on shared outcomes across campaigns; and (3) to be comprehensive, the evaluation should assess the effectiveness and efficiency of campaigns. If the research team had designed an evaluation under a different set of assumptions (e.g., focusing on the most influential pieces of each campaign, rather than shared elements across campaigns), it might not have come to the same conclusions. This raises the issue of precisely which assumptions should guide a cross-agency evaluation of this type. Contemplating this issue leads to broader, overarching questions: What should each agency's role be in promoting greater access to care for service members and veterans? To what degree should efforts overlap or be unique? Continuing to take a cross-agency approach to answering these questions and understanding the success of campaigns individually and collectively will be important to improving mental health outcomes for service members, veterans, and their families. Because of a major reorganization of the Defense Health Agency, publication of this report was significantly delayed. However, improving mental health outcomes for these populations remains a priority, despite the fact the that the CAP-G and associated interagency task force and working group that commissioned this report no longer exist.