An Evaluation of the Implementation and Perceived Utility of the Airman Resilience Training Program

Background

Since 2001, the U.S. Military has been functioning at an operational tempo that is historically high for the all-volunteer force. Even with the 2011 drawdown of troops in Iraq, servicemembers have been experiencing stressful and frequent deployments, which are increasing stress and strain on relationships and the general psychological well-being of servicemembers and their families. In response to these challenges, the U.S. Department of Defense (DoD) has implemented numerous programs to support servicemembers and their families in this area. In 2009, the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury asked RAND to help identify and examine the effectiveness of DoD-sponsored programs designed to support servicemembers and their families. One such program is Airman Resilience Training (ART), developed by the Air Force's Office of Deployment Psychological Health and implemented in November 2010. It replaced its predecessor, Landing Gear, which had been in place since April 2008.

ART is a psycho-educational program that aims to provide deploying airmen with tools and techniques to improve their ability to cope with stressful events and to facilitate their smooth reentry into work and family life upon returning from deployment. It is delivered in a workshop or class setting and consists of a set of PowerPoint briefing slides and accompanying training manual. The briefing provides information to the briefer on the content of the slide and recommends types of examples or illustrations to share with the audience. ART can be delivered in a variety of locations; briefers are allowed to be flexible in their approach; and the installation can determine the frequency and scheduling of classes based on the airmen's deploying and reintegrating schedules. All Air Force deployment sites (installations or bases that deploy airmen) are required to provide resilience training to their deploying and reintegrating airmen. ART fulfills this requirement. Although ART is the Air Force's official resilience training briefing, installations are allowed to use any other resilience training program that suits the needs of their deploying airmen, but ART is required for reintegrating airmen.

Study Objectives

This study had two objectives: (1) to ascertain the extent to which ART was being implemented according to its original design and (2) to gauge its potential usefulness and value as perceived by deploying and reintegrating airmen and mental health professionals. This study does not directly evaluate the program's effectiveness in promoting resilience. Instead, the study is an implementation evaluation, which aims to describe how ART is being implemented and to provide insight into its potential to meet its intended goals of improving resilience, reducing stress, and improving help-seeking behavior among airmen.

Methodology

To fulfill the study's objectives, the RAND team employed a case study design in which we observed the delivery of ART and conducted interviews and discussion sessions in four Air Force installations that utilize ART. From August 2011 through November 2011, RAND conducted site visits to assess how ART was implemented in four deployment centers and to document how useful airmen and mental health professionals at those installations consider ART.

Rationale for Site Selection

With the assistance of staff in the Air Force's Office of Deployment Psychological Health, RAND selected four installations to be used as sites for this study. These sites were selected based on a number of strategically chosen criteria to ensure that we captured diversity in installations' (1) location in the United States, (2) organizational function or mission (defined as Major Command), and (3) size of military population. We opted to use data-driven purposive sampling in order to include cases that would reveal a variety of possible implementation scenarios, as opposed to random sampling (Stake, 1995).

Data Collection Activities and Analytic Approach

We collected data during our site visits through three activities: (1) structured observations of ART briefings, (2) discussion sessions with deploying and reintegrating airmen after they were briefed, and (3) interviews with the installation's chief mental health provider and his or her staff.

To capture the variation in how ART was implemented across the four sites and perceptions of ART's usefulness for discussion session participants and mental health staff, we summarized information gathered from discussion session conversations and interviews with mental health staff at each site and then coded the information along major themes of interest, noting any consistencies or differences across the sites. We are cautious not to generalize our interpretation of the data beyond the four sites that participated in the study.

Findings

Implementation of ART Varied Across and Within the Sites

We found that ART was provided to airmen in the same setting (within the deploying and reintegration checklist process), but that its delivery varied. Some briefers followed the slides closely and some exemplified ART points or content with relevant statistics or anecdotes, as recommended in the training manual. No ART sessions included significant participant discussion as recommended in ART instructions, no airmen ever asked any questions, and most airmen appeared disengaged with the briefings, as exemplified by distracted behaviors, including texting on cell phones, closing eyes and putting their heads on tables or desks, and occasional chatting among peers.

By design, the members of the Mental Health Office that brief ART are allowed discretion in how to deliver ART depending on the needs of the airmen in the audience. However, across our sites, the ART briefings were delivered differently not based on the airmen's purported needs, but for the following reasons:

• Institutional setting: There was significant variation in the location and facilities where the briefing was provided and in the time allotted to ART by organizers of the deployment and redeployment checklists. In some instances, the ART slides were not available or the room where ART was to be briefed did not allow for PowerPoint presentations. On most occasions, briefers were only allotted five to ten minutes, although the instructions that accompany ART recommend that briefers take 30–60 minutes, and up to three hours for a high-needs audience.
• Briefers' characteristics: The briefers' deployment experience, mental health training, and experience with public speaking or leading discussions all varied across the sites, affecting the delivery of the material and the type of information that airmen were exposed to.

Perceived Usefulness of ART Was Generally Low

Discussion session participants reported that they did not recall much content from the ART briefing and most reported that they did not consider the information provided in ART to be useful for promoting their resilience or reintegration. They suggested several reasons for this:

• Briefing fatigue: The ART briefing was delivered to airmen in tandem with a long list of required briefings, many of which occurred on the same date or contiguous with ART.
• Perceived redundancy with other programs that intend to promote resilience, such as suicide prevention training or Wingman Days: Many of our discussion session respondents felt that they were inundated with the information presented in ART. In turn, they reported “tuning out.”
• Format as a PowerPoint presentation: Teaching resilience skills through a set of briefing slides did not seem to encourage active learning of concrete coping skills, but rather the passive absorption of information. The format of ART as a PowerPoint slide deck also reportedly discouraged active participation and discouraged the dissemination of practical skills to airmen.
• Lack of tailoring: ART was presented to all audience members in each session in the same way, without recognition that each audience included airmen with different deployment experiences, missions, or combat experience. By design, ART briefers could adjust the briefing to fit the audience's needs, yet this is impossible to do when the audience has a wide mix of airmen. In practice, in the sites we visited, ART is delivered in a one-size-fits-all approach.
• Content could be improved: Informants reported that the information provided within the slides was often vague, and the specific behavioral coping skills (i.e., guidelines pertaining to receiving proper nutrition and sleep) and topic areas (i.e., maintaining healthy relationships with family members) they view as important were not covered.

Suggestions for Improvement

The study's findings suggest that at the four sites, there may be opportunities where ART can better meet its intended goals to promote the resilience of deploying airmen and to support the smooth reintegration of airmen returning from deployment. Given these findings, we suggest two ways that the Air Force could improve ART. Given the limited empirical evidence for resilience-building tactics or programs, we do not make suggestions for how the Air Force should conduct resilience training, but rather suggest that the Air Force focus on internal quality improvements by (1) conducting a needs assessment to identify the most appropriate goals for ART as well as an assessment to identify overlapping efforts within the Air Force on resilience training and (2) consider modifying ART's content and delivery based on our study participants' suggested strategies.

Conduct Assessments to Identify Appropriate Goals for ART

Given our findings that many airmen and mental health professionals reported that the ART session was not considered very useful and was possibly redundant with airmen's prior training or other briefings, we suggest that the Air Force conduct two types of assessments to best determine the goals, content, and structure of ART that will ensure that it is providing appropriate content in an effective and efficient manner. First, a needs assessment may not only gauge the current gaps in resilience training for airmen but may also identify a more targeted audience who may be at higher risk for experiencing stress while in theater or reintegrating into work and family life. If areas of resilience training most pertinent to these airmen are identified, the assessment could help determine how ART should address the needs of servicemembers who may benefit most from the training. Once the array of airmen's needs is determined, the Air Force could conduct another assessment to ascertain the extent of overlap in the content of resilience-related trainings currently provided by individual installations, Major Commands, and across the Air Force. The goal of such an assessment would be to redesign ART to be less redundant with other efforts and to repeat training material only when repetition is needed to improve learning or for specific groups that are at a high risk of encountering psychological health issues.

Implement Strategies for Ongoing Quality-Improvement of ART

If the goals of ART remain the same as they are now, we offer some suggestions for improving upon its current content and delivery as part of ongoing quality improvement efforts. We suggest ways the Air Force might improve ART to better meet the needs of deploying and reintegrating airmen, recognizing the variety of deployment and combat-related experiences across the Air Force. We base our suggestions for improvement on the findings from our site visits, on the evidence base for program effectiveness, and the limited scientific literature on resilience. Before deciding to modify ART, it is important for the Air Force to consider the feasibility of implementing any changes.

Modify the Content of ART

Suggestions for improving the content of ART are based on the perceived needs and impressions of airmen who participated in the interviews and discussion sessions in this study and the limited scientific literature on resilience training. Since there are no empirically demonstrated resilience-building tactics or programs that provide strong scientific guidance regarding modifications to content, these suggestions should be seen as part of a process of internal quality improvement.

• Design the content to meet the needs of specific intended audiences (e.g., first-time deployments, deployments to combat areas) and tailor the content and language to the challenges facing that audience. While ART is designed to be modified by the briefer to adapt to the audience, the actual setting does not allow such tailoring, as the audience usually contains a diverse mix of servicemembers. Also, the briefer has no prior knowledge of who will be attending the training session, further circumscribing his or her ability to plan ahead.
• Focus on skills training in ART. Focus on teaching a few specific, concrete skills or coping behaviors that airmen are likely to use, rather than delivering a broad educational course on the determinants of resilience. This new, narrow focus may require skill demonstration, practice, and detailed examples of when the behavior may be useful.
• Allow airmen more choice in the resilience training they receive. Consider allowing airmen to select among required resilience courses that will best prepare them for their expected challenges during deployment. For example, courses on specific deployment-related skills, such as “sleep and nutrition,” “reducing stress and relaxation (antianxiety),” “maintaining energy and focus (antidepression),” and “effective parenting from abroad.” Airmen could complete their required resilience training by selecting any of the available courses, and those with multiple deployments could select different content for each deployment to be less repetitive. To ensure that airmen are receiving necessary skills, they could make the selection in consultation with the installation's mental health office, or the Air Force could mandate a minimum number of courses or types of courses be taken in a year.
• Incorporate engaging anecdotes and examples in a standardized way. Given the variation in the type and utility of anecdotes or illustrations provided by the briefers we observed in this study, consider all or part of ART as a videotaped presentation by an expert presenter with well-selected anecdotes. Someone from the mental health office can still be available for facilitating discussion and answering questions. It is difficult to place all responsibility solely on the briefers to consistently supply engaging material when circumstances are wide ranging regarding amount of preparation time, presentation setting, and background experience.

Modify the Delivery and Implementation of ART

Delivery of ART (as currently structured) could be improved to ensure that airmen are more attentive to the training and therefore more likely to fully absorb and process its content. We draw these suggested improvements from the perceptions, experiences, and impressions from airmen in our site visits as well as from the established research on program implementation.

• Ensure buy-in from Air Force personnel who are involved with implementation and delivery. We found that organizers of the deployment and reintegration process at the sites we visited expected ART to conform to their scheduling constraints. The disconnect between ART developers' expected duration of 30–60 minutes and deployment and reintegration process organizers' expectations led to some tense situations. If ART is to remain a part of the deployment and reintegration process, installation commanders need better communication with briefing organizers. Without buy-in from those who are part of the implementation process, a program will not succeed in meeting its goals.
• Recalibrate the scope of material covered or the timing allowed. A clear hindrance to the briefers' ability to provide ART as designed was the lack of time and facilities. In practice, briefers are delivering content designed for a 30–60 minute presentation in about 10 minutes. It may be helpful if either (1) the briefers are required to spend more time delivering ART (and this requirement should be clearly communicated to the organizers of the deployment and reintegration processes) or (2) ART is scaled back so that the content can be usefully covered in much less time. If potential redundancies with other Air Force resilience training programs are identified in the assessment suggested above, then it may be possible to scale back the content of ART without airmen losing skills training.
• Minimize the extent to which the resilience training takes staff resources away from treatment activities. Incorporating computer-based training, a video module, or using briefers who are not treatment providers might help with this goal.
• Institute criteria for who should brief ART. To limit the variation in briefing style or perceived legitimacy, one option is to specify criteria to ensure that briefers have experience or skills in presenting briefings, or that they have deployment experience. As is, ART instructions recommend that briefers be mental health or Integrated Delivery System personnel, with no recommendations for deployment experience or the type of training the briefer should have to deliver presentations. For example, it may be helpful to avoid having a servicemember with no deployment experience briefing those with combat experience on handling deployment stress. Furthermore, it may be more important that the briefer have a diverse deployment history than that the briefer be a mental health treatment provider.
• Ensure that briefers receive clear guidance and training on how to deliver ART and what content to cover. To ensure that briefers are adequately and appropriately trained to deliver ART, more than written instructions need to accompany the briefing slides. When ART was first launched, the Office of Deployment Psychological Health held teleconferences and webinars to describe the program. One option is to provide these types of information sessions on a regular basis for new briefers. Another option is to hold in-person training sessions at a centralized location for all briefers. A third option is to utilize personnel already trained in resilience training, such as Master Resilience Trainers, once that program fully stands up.
• Reconsider using PowerPoint slides as the primary medium. To better engage airmen, one suggestion is to not have ART delivered solely in a PowerPoint format because this format promotes passive learning. If more time is allotted to the briefers, one option would be to incorporate more hands-on learning experiences for airmen, which would encourage more active listening. Such experiences may include role-playing, games, or tactics that may involve interacting with fellow airmen in the room.
• Track implementation. For any program to be successful in meeting its intended goals, it is important to measure its delivery or implementation. If ART briefers diverge from delivery approaches articulated in the instruction manual, then there is little chance that the program will meet any of its intended goals. To determine whether installations are implementing ART as it is designed, the Office of Deployment Psychological Health could administer surveys to briefers that measure what is known as “fidelity of implementation.” These surveys should have questions that allow briefers to gauge the extent they are able to deliver the briefing as designed and to express facilitators or barriers to their being able to deliver the briefing. ART developers can make changes to the program, if needed, and can gauge any potential areas of weakness based on this information.

Conclusions

These suggestions could improve the Air Force's resilience training by creating a program that is more engaging, skills-focused, targeted to those at risk, and memorable, while placing less of a workload on mental health care providers.

Reference

Stake, R. E., The Art of Case Study Research, Thousand Oaks, Calif.: Sage Publications, 1995.

The research described in this article was prepared for the Office of the Secretary of Defense (OSD). The research was conducted within the RAND National Defense Research Institute, a federally funded research and development center sponsored by OSD, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.