Evaluating Public Awareness Campaigns to Promote Military and Veteran Mental Health

Photo by Anna-Victoria Crisostomo/U.S. Navy

When service members return from conflicts abroad, some bring with them depression, posttraumatic stress disorder, substance use disorders, and other mental health issues that can affect their quality of life, their ability to function at home and on the job, and their relationships with family and friends. Many do not realize that they need help, or they hesitate to seek treatment because they fear that it will have negative repercussions on their careers. Without care, these service members and veterans may face lifelong challenges.

In 2015, the Obama administration designated improving mental health outcomes for service members, veterans, and their families as a Cross-Agency Priority Goal (CAP-G). In response, the Executive Office of the President, U.S. Department of Defense (DoD), U.S. Department of Veterans Affairs (VA), and U.S. Department of Health and Human Services (HHS) commissioned an evaluation (funded by the DoD Psychological Health Center of Excellence) of several mental health public awareness campaigns to better understand the degree to which they help people overcome negative perceptions of mental health disorders and treatment-seeking and promote awareness of mental health resources.

With the goals of better understanding whether these campaigns are working and identifying ways to improve the federal approach to this issue, an interagency task force asked RAND to conduct an evaluation of four such efforts: the Real Warriors Campaign (DoD), Make the Connection (VA), outreach efforts of the Veterans Crisis Line (VA), and Recovery Month (HHS). This brief describes the results of the cross-agency evaluation, which examined the scope, content, and dissemination of the campaigns.

About the Evaluation

The RAND team's cross-agency evaluation sought to understand the campaign's collective scope, content, and dissemination by analyzing campaign processes, material, and social media data. The cross-agency evaluation was designed to address goals, messages, and processes shared across multiple agencies and campaigns, so the results presented here are not intended to serve as a comprehensive evaluation of each campaign, its effectiveness, or its cost-effectiveness.

Key Findings and Recommendations

The following sections summarize the RAND team's findings and recommendations in four areas: efficiency and mental health messaging, content of campaign materials, dissemination and reach, and inclusion of resources that directly connect individuals to care. At the end of each recommendation, we note in parentheses who should take the lead in addressing that issue—a federal partner group (such as an interagency task force or working group) or the campaigns themselves.

Efficiency and Mental Health Messaging

The four campaigns, which have distinct yet overlapping audiences, share targeted outcomes (to improve knowledge of mental health symptoms and create positive perceptions of individuals with mental health conditions and of mental health treatment). They offer a variety of messages—for those who are at risk of mental illness, considering treatment, in need of immediate support, or in recovery—to their common audiences and unique messages when appropriate.

A majority of campaign content (82 percent across all four) aligns with shared goals, and the individuals pictured in the materials reflect the target audiences of each campaign. The campaigns could coordinate more with each other and their shared partners, and could cross-reference each other more as resources.

Because campaigns were initiated at different points in time, no shared measures were being used across campaigns to facilitate tracking across overlapping target populations.

Recommendations

1. Reference other campaigns as resources in materials (campaigns).
2. Convene a set of national and regional partners to help define a strategic national direction for public mental health education that DoD, VA, and HHS can use to guide their efforts (federal partner group, working with the campaigns).
3. Clearly define the unique contributions and intentional overlap in messages and target populations across the campaigns (federal partner group, working with the campaigns).
4. Monitor coordination of current and new campaign efforts and promote coordination of messaging and dissemination (agencies, working through federal partner group).
5. Develop a targeted set of performance measures that cuts across campaigns (federal partner group).
6. Explore the development of cross-campaign and cross-platform measures to fully capture exposure and synergy among campaigns (federal partner group).

Content of Campaign Materials

Most campaign content aligned with best practices; clearly communicated the messages; and used credible, positive messengers. All of the campaigns use testimonials from individuals about their own experiences with mental illness, substance use, and recovery, which can help reduce negative perceptions of people with mental illness and increase positive perceptions.

However, the campaigns' content could be improved in several areas. Half of the campaign websites did not cite the sources of the information used on webpages, and some campaign content did not align with a specific campaign outcome. Campaign sites housed some faulty links or outdated content that should be updated. Some experts who reviewed the campaigns thought that they could be clearer about how service members' anonymity or confidentiality is protected. In addition, campaigns that tried to serve multiple audiences often had content for secondary audiences (e.g., family members and health professionals) that the expert panel found limited and less developed than content focused on each campaign's main audience.

Recommendations

1. Determine whether source information should be clearly marked on more campaign materials (campaigns).
2. Review campaign content that does not align with a specific campaign outcome (or shared outcome) and consider modification or deletion (campaigns).
3. Review website content and links to ensure that they are all current (campaigns).
4. Specify the level of anonymity or confidentiality guaranteed by self-assessment and direct connections to mental health care (e.g., call lines, chat lines, direct connections to a local medical center) (campaigns).
5. Enhance materials targeting support networks (and other target audiences, as relevant)—but also consider whether to serve a smaller audience and develop more and richer content for the main target audience of service members and veterans (campaigns).

Dissemination and Reach

Campaigns used multiple approaches to disseminate their materials and messages, including websites, public service announcements (PSAs), social media, television, radio, and in-person engagement. While the reach of all campaigns (except for Recovery Month) increased between 2012 and 2015, it was impossible to measure the degree to which campaigns reached their specific target audiences because campaigns were not collecting that data.

The analysis found that the campaigns reached as many as 5.6 million people in 2015 through Facebook, YouTube, and Twitter. VA's Make the Connection campaign had the greatest reach on these platforms, providing more than 90 percent of the campaigns' collective Facebook fans and YouTube views. Although campaigns used Twitter in a very limited way, they still managed to connect with users who have above-average influence in dissemination of tweets about mental health.

The campaigns' websites played host to more than 4 million visitor sessions. However, a significant share of visitors spent only a brief amount of time on the websites and had minimal engagement.

The campaigns aired more than 400,000 radio and television PSAs, resulting in more than 42.9 billion impressions (the number of audience members who might have been exposed to the PSA). Ninety percent of impressions came from DoD's Real Warriors Campaign. The campaigns also distributed more than 10 million campaign brochures, pieces of merchandise, and other items through in-person outreach at more than 250 events and by partnering with more than 700 organizations and agencies. The timing of outreach varied, with some campaigns highly active during certain time periods (particularly Recovery Month). It is unclear to what degree campaigns coordinated the timing of their activities.

Recommendations

1. Develop strategies to use Twitter more effectively (campaigns).
2. Modify campaign websites to be more engaging and meet the needs of brief visitors (campaigns).
3. Consider whether the campaigns should intentionally time their active outreach to occur in different months to increase the likelihood of reaching overlapping target populations or to time outreach at the same time of year to achieve more saturation in a shorter period (campaigns working through federal partner group).

Inclusion of Direct Connections to Care

Although all campaigns offered general resources on mental health problems and treatment (e.g., locator tools that allow users to find care nearby, outreach phone lines that provide general resources), just 27 percent of campaign materials provided direct connections to care (defined as resources that connect to a crisis chat or phone line or connect directly to a specific medical center to make an appointment).

Providing direct connections to care may be a difficult task, but this approach is relevant to the campaigns' shared goal of encouraging target audiences to seek care if needed.

Recommendations

1. Ensure that a direct connection to mental health care (e.g., phone line, live chat line, direct connection to a local medical center) is included on relevant campaign materials (campaigns).
2. Determine whether campaigns should offer a centralized call line that allows people who are not in immediate crisis to make an appointment for mental health care. Although this is not feasible for any individual campaigns to execute, this direct connection to care is strongly encouraged to remove barriers to treatment for service members and veterans (federal partner group).

Conclusion

The cross-agency evaluation of the four DoD, VA, and HHS mental health public awareness campaigns detailed in this report relied on three assumptions: (1) To align with CAP-G, the evaluation should focus on service members, veterans, and their families; (2) to be efficient, the evaluation should focus on shared outcomes across campaigns; and (3) to be comprehensive, the evaluation should assess the effectiveness and efficiency of campaigns. If the research team had designed an evaluation under a different set of assumptions (e.g., focusing on the most influential pieces of each campaign, rather than shared elements across campaigns), it might not have come to the same conclusions. This raises the issue of precisely which assumptions should guide a cross-agency evaluation of this type. Contemplating this issue leads to broader, overarching questions: What should each agency's role be in promoting greater access to care for service members and veterans? To what degree should efforts overlap or be unique? Continuing to take a cross-agency approach to answering these questions and understanding the success of campaigns individually and collectively will be important to improving mental health outcomes for service members, veterans, and their families. Because of a major reorganization of the Defense Health Agency, publication of this report was significantly delayed. However, improving mental health outcomes for these populations remains a priority, despite the fact the that the CAP-G and associated interagency task force and working group that commissioned this report no longer exist.