Evaluating an Operator Physical Fitness Test Prototype for Tactical Air Control Party and Air Liaison Officers

In January 2013, then–Chairman of the Joint Chiefs of Staff Martin Dempsey and then–Secretary of Defense Leon Panetta issued a memorandum rescinding the 1994 Direct Ground Combat Definition and Assignment Rule, which excluded women from assignment to units and positions whose primary mission is to engage in direct combat on the ground. In the memorandum, Panetta and Dempsey mandated that "[c]urrently closed units and positions will be opened by each relevant Service … after the development and implementation of validated, gender-neutral occupational standards and the required notification to Congress" (Chairman of the Joint Chiefs of Staff, 2013). To comply with this mandate, the U.S. Air Force, with the assistance of the RAND Corporation, established a process in fiscal year (FY) 2012 to identify and validate gender-neutral tests, standards, and physical requirements. This initial effort was followed with additional technical support by RAND in FY 2014 and 2015 for the Air Force's implementation of an extensive criterion-related validation study based on scientific principles. This study led to the recommendation of a ten-component Operator–Prototype Test Battery (O-PTB) for the enlisted Tactical Air Control Party (TACP) specialty and the Air Liaison Officer (ALO) specialty.¹ A list of these tests is provided in Table 1.

The AF-ESU is now leading efforts to implement these tests. As part of this effort, the AF-ESU developed an implementation, verification, and training (IVT) plan to address several questions such as:

• Who will administer the tests?
• How long will it take to administer and take the tests?
• What is the likelihood of a test-taker sustaining an injury while taking the tests?
• How many current TACPs and ALOs would be expected to pass the proposed test standards?
• How much improvement can be expected in test performance as TACPs and ALOs become more familiar with the tests?
• How well does performance on the test battery differentiate between successful and less successful TACPs and ALOs?
• What concerns do the different stakeholders have about the tests and standards?

Purpose and Approach of the Research

Even though the AF-ESU prioritized these questions, other short- and long-term issues and concerns should be identified and may need to be subsequently addressed. Consequently, RAND was asked by Air Force Director of Military Force Management Policy, Deputy Chief of Staff for Manpower, Personnel and Services to offer support by designing a preliminary evaluation effort of the implementation of the physical tests and standards being adopted by the TACP and ALO career fields. Our evaluation emphasizes one of the main IVT questions: the concerns of different stakeholders. After reviewing initial results from this evaluation, we developed a broad evaluation framework to identify other possible issues and concerns that may emerge in relation to the implementation of physical tests and standards.

Our approach to achieve these two objectives consisted of four steps:

1. Identify relevant stakeholders.
2. Design evaluation instruments to address stakeholder reactions.
3. Collect and analyze the data.
4. Develop an evaluation framework for future evaluation efforts.

Step One: Identify Relevant Stakeholders

The Air Force plans to sequence the tests and standards implementation in three waves that correspond to three primary job roles for TACPs and ALOs. Specifically, as tests and standards are adopted, they will first be implemented for TACP and ALO operators, then technical training students, and finally for recruits. At this time, the Air Force is in its first wave of implementation (for TACP and ALO operators). Therefore, the most immediate priority for our preliminary evaluation efforts is to determine stakeholder groups involved in these initial implementation efforts for TACP and ALO operators. Three primary stakeholder groups were identified:

• TACP and ALO operators, who will be required to take the tests and meet the specified standards
• physical training leaders (PTLs), who will be responsible for administering and scoring the tests and providing training to other test administrators
• career field managers (CFMs), who will be responsible for addressing gaps in readiness levels for their specialty and for overseeing whether resource needs are being met across the career field.

Step Two: Design Evaluation Instruments to Address Stakeholder Reactions

Next, we identified several topics that could affect the successful implementation of the O-PTB. Taking into consideration the specific topics most relevant to each stakeholder group and prior research on test-taker reactions, we identified the following primary topics for each stakeholder group:

• operator perspective (TACPs and ALOs)
  • consistency of test administration
  • knowledge of test performance relative to test standards
  • injury concerns related to the tests
  • experienced levels of frustration in taking the tests
  • global evaluations of the O-PTB, including perceived utility, validity, and fairness
• PTL perspective
  • quality of training provided to administer and score tests
  • global evaluations of the O-PTB, including perceived utility, validity, and fairness
• CFM perspective
  • current and future plans for test implementation
  • perceived benefits of implementing a new test battery
  • perceived challenges or drawbacks of implementing a new test battery
  • concerns with the specific recommended test battery
  • specific barriers and concerns for test implementation such as time to administer, potential for injury, cost, fairness, and utility.

To address these topics, we used a mixed-methods approach. Specifically, we developed evaluation surveys for TACP and ALO operators and for PTLs and conducted semistructured interviews with CFMs.

Step Three: Collect and Analyze the Data

Evaluation surveys were administered to TACPs, ALOs, and PTLs during the AF-ESU implementation trips to different installations. For each implementation trip, the AF-ESU trained the PTLs on how to properly administer and score each test in the O-PTB and then administered tests to a sample of TACPs and ALOs available at the time of the trip.

Evaluation surveys were completed by a total of 198 operators (TACPs and ALOs) and 135 PTLs representing units from 12 different installations. We also conducted semistructured interviews with CFMs both from the TACP and ALO specialties, as well as other BA specialties including Pararescue (PJ), Combat Rescue Officer (CRO), Special Tactics Officer (STO), and Special Operations Weather (SOWT).

Step Four: Develop an Evaluation Framework for Future Evaluation Efforts

Our initial evaluation is focused on topics relevant to immediate implementation priorities of the Air Force, however, we recognize a broader perspective on evaluation is beneficial to identify potential future concerns or issues that emerge as priorities over time (e.g., focus on tests and standards for recruits). Consequently, we present a more comprehensive framework identifying a range of possible topics that are organized in a framework often used in the military for identifying requirements and potential gaps for a given set of strategic objectives.² Broadly, the objectives of the evaluation framework are to (1) raise awareness of potential challenges and concerns for relevant stakeholders during the implementation and adoption of the new physical tests and standards and (2) promote the development of systematic data collection to monitor progress over time.

Results of Evaluation Surveys and Semistructured Interviews

In the following sections, we present an overview of results from each of the stakeholder groups. We begin by discussing results from the evaluation surveys from the TACP and ALO operators and then PTLs. We conclude with a summary of themes identified during the discussions held with CFMs.

TACP and ALO Survey Responses

Consistency of Test Administration

The pattern of TACP and ALO responses suggested that each test was administered consistently. Specifically, over 90 percent of respondents agreed or strongly agreed that each test was consistently administered. Furthermore, only two tests, Pull-Ups and Extended Cross Knee Crunch, had less than 95 percent agreeing or strongly agreeing that these tests were administered consistently (94 and 92 percent, respectively).

Knowledge of Performance Relative to Standard

TACP and ALOs indicated that they knew how well they performed on each test relative to the required standard established as part of the Air Force validation study. Only the Extended Cross Knee Crunch yielded less than 90 percent of responses agreeing or strongly agreeing, with 9 percent of respondents indicating they neither agreed nor disagreed with the statement.

Injury Concerns for Each Test

Overall, TACP and ALO respondents noted few injury concerns across the majority of the tests, and no new injuries were reported in preparation for or during administration of the tests. However, 20 percent of respondents indicated an injury concern for the Trap Bar Deadlift. Additional analyses indicated no patterns in responses for the Trap Bar Deadlift across subgroups. However, respondents' open-ended comments suggest concerns primarily emphasized the ability to use proper form and technique when performing the lift. Some of the respondents provided specific suggestions to address potential injury concerns, including providing sufficient training on proper form, providing opportunities to practice, and using a weight belt when lifting.

Frustration Experienced

Tests that produced the most frustration included the Extended Cross Knee Crunch (41 percent) and the Medicine Ball Toss (21 percent); these tests demonstrated significantly more frustration compared with other tests in the test battery. We conducted additional analyses for these two tests to determine if any significant patterns emerged by installation and background characteristics.

Although installation was not influential, comparisons of responses by demographic variables revealed that TACPs and ALOs who are heavier, taller, and more experienced were significantly less likely to indicate frustration on the Medicine Ball Toss compared with lighter, shorter, and less experienced TACPs and ALOs. This finding is further supported by a few open-ended comments suggesting that the Medicine Ball Toss appears to favor taller individuals. Other common open-ended comments indicated that some respondents felt that the Medicine Ball Toss required substantial technique and practice to do well, with some stating that the test measures skill/technique more than strength/power.

Global Evaluations of the Test Battery

Several items were used to address operators' perspectives on the perceived utility of the test battery, in addition to perceptions of how well they think the test battery measures important abilities for their job (i.e., face validity). Overall, the general perceptions suggest that the test battery will be positively received by TACPs and ALOs, with 84 percent of respondents indicating that knowing how well they performed on this test battery will help improve job-related physical capabilities; 74 percent indicated that this test measures the abilities required of a TACP.

Although most responses present favorable perceptions, 18 percent indicated that they disagreed or strongly disagreed that the test would be fair for all TACPs regardless of rank, age, stature, gender, or race/ethnicity. Respondents' open-ended comments suggested that the tests and standards might not be relevant to all TACPs given that some assignments do not require any (or as much) physical effort to perform their job tasks.

PTL Survey Responses

PTL Training

We also developed survey items for the PTLs who received test administrator training from the AF-ESU on how to properly administer and score each test. Overall, the PTL responses to the training clearly suggest that the training provided was effective.

Global Evaluations of the Test Battery

PTLs also provided responses to general questions about the prototype test battery. Overall, these responses were also favorable. Ninety-three percent of respondents agreed or strongly agreed that each test was administered to all operators in the same way. Eighty percent of PTLs indicated that the test battery would be fair to all TACPs regardless of rank, age, stature, gender, or race/ethnicity. Some of those who indicated concerns about fairness provided open-ended comments suggesting that fitness requirements may not be relevant across the entire career lifecycle of a TACP. That is, TACPs may be assigned to a position in which fitness is less important, such as a staff position. Other open-ended comments suggest that some of the frustration observed in taking specific tests (e.g., Extended Cross Knee Crunch) may be related to lack of familiarity in taking a new test and a desire to perform well.

Perceived Utility of the Test Battery

The final set of survey items for PTLs focused on the overall utility of the test battery. Responses indicated very positive support with 97 percent agreeing or strongly agreeing that the test battery is a better measure of operational capabilities compared with the Physical Ability and Stamina Test (PAST).³ Furthermore, 94 percent agreed or strongly agreed that operators could really show their physical abilities through this test battery. And, 92 percent indicated that TACPs would find their test results useful for improving their job-related physical capabilities.

CFM Discussions

To gain a better understanding of the BA CFMs' concerns about test implementation, we held three separate meetings with individual and small groups of CFMs, which included CFMs and career field representatives from TACP and ALO; CRO, PJ, and STO; and SOWT. One CFM (Combat Control) was unavailable for these meetings. Although only the TACP and ALO specialties have moved forward with implementation of a new operator test battery, we decided to include other BA CFMs to provide a more comprehensive understanding of the issues, concerns, and barriers influencing the successful implementation of an updated physical test battery. Given the differences in current and planned efforts to implement an updated physical test battery, we present our discussion of the CFM feedback for each topic area grouped by TACP/ALO CFMs and other BA CFMs. Several questions were presented to address the following topics:

• current and future plans for test implementation
• perceived benefits of implementing a new test battery
• perceived challenges or drawbacks of implementing a new test battery
• concerns with the specific recommended test battery
• specific barriers and concerns for test implementation such as time to administer, potential for injury, cost, fairness, and utility.

Overall, the CFMs agreed that there are potential benefits gained from the proposed O-PTB; however, other BA CFMs raised concerns about the need for ten tests, the time to administer them, and the potential costs associated with purchasing and maintaining the tests (see Table 2).

Conclusions

We conducted a preliminary evaluation of potential issues and concerns that may influence implementation effectiveness for TACP and ALO operators and developed a broader framework for monitoring and evaluating the implementation of an occupationally specific PF test battery for TACPs and ALOs. A summary of the main findings and recommendations is provided in Table 3.

Overall, the results from TACPs, ALOs, and PTLs were very positive, with relatively few concerns identified among a minority of TACP and ALO operators who participated in this phase of test implementation. Most indicated that each test was administered correctly and that the test battery measured important job-related physical abilities. Many open-ended comments further supported these findings with positive comparisons to the PAST, which stated that the O-PTB is more comprehensive and more representative of operational tasks than the PAST. Only a few tests caused frustration or raised concerns for potential injury among some of the respondents. Three tests, in particular, warrant close monitoring and further review to identify opportunities to address potential concerns for injury and frustration: the Extended Cross Knee Crunch, Trap Bar Deadlift, and Medicine Ball Toss. For each of these tests, we recommend additional training, practice, and feedback on the proper form and technique, which should help address these concerns.

Analysis of open-ended comments in combination with CFM feedback suggests that additional structured communication could help address a range of concerns (e.g., career repercussions if operators fail a test) and to further educate the operators on the purpose of the test battery and the steps that were taken to inform the decision to select the final ten tests and standards. Consequently, we recommend developing additional communication channels such as short, informative pamphlets that can be used to answer frequently asked questions. For example, operators could benefit from additional education on the role of each test in assessing their physical capability to perform critical physical tasks of their specialty. Operators could also benefit by having additional information on the implementation plan, timeline, and potential changes that may be made to the protocols for specific tests (e.g., Farmer's Carry).

Although the results were very positive, our analysis represents only a snapshot of perceptions at a given point in time. As the implementation proceeds for TACP and ALO operators, we recommend following up with future evaluations to determine if perceptions have changed. Future evaluations should be conducted annually for the first two years of implementation and then every three to five years thereafter. Tracking individual operators over time using pre- and post-test research designs will allow for more sophisticated analyses for evaluating the effectiveness of different interventions (e.g., additional training on form/technique) to address concerns. Additional interviews, focus groups, and surveys of stakeholders can also help to determine whether new issues or concerns have emerged.

Finally, we recommend expanding these evaluation efforts to consider additional stakeholder groups and topics. To assist in identifying other stakeholder groups and topics, we developed a framework guided by the DOTMLPF-P structure. This framework could be used to raise awareness of possible issues that may influence the successful implementation of PF tests and standards and to guide evaluation efforts for determining how well implementation objectives are being met.