The Department of the Air Force maintains rich records on the knowledge, skills, abilities, and other attributes of its personnel. To standardize and extract meaning from these textual records, the authors reviewed state-of-the-art natural language processing and machine learning approaches. They applied these approaches to text from officer performance reports and created a system called the Personnel Records Scoring System (PReSS).

The Personnel Records Scoring System

Volume 3, A Methodology for Designing Tools to Support Air Force Human Resources Decisionmaking

David Schulker, Joshua Williams, Cheryl K. Montemayor, Li Ang Zhang, Matthew Walsh

ResearchPublished Feb 27, 2024

The Department of the Air Force (DAF) maintains rich records on the knowledge, skills, abilities, and other attributes of its personnel. The personnel system records much of this information as structured lists and free-form text, which creates a need for many processes in which human resources decisionmakers must review the records by hand before issuing a decision. These processes become more difficult with larger populations of officers, as any human judge faces capacity limitations. Thus, there is an opportunity for artificial intelligence applications to improve the quality of inputs for these review processes, helping the human resources management system to become more effective and/or more efficient in meeting DAF strategic goals.

To develop a computational approach to standardize and extract meaning from textual records, the authors reviewed state-of-the-art natural language processing (NLP) and machine learning (ML) approaches. They applied these approaches to text from officer performance reports (OPRs) and used them to predict O-5 and O-6 promotion outcomes and Developmental Education Designation Board scores. The resulting system created for this research is called the Personnel Records Scoring System (PReSS). Aside from the use of ML models to predict board results for future candidates, the authors propose a methodology for using models to generate summary reports that highlight the most-significant statements and detractors contained in a service member's records.

Key Takeaways

  • DAF analysts can rapidly develop simple models relating key text in officer records to past decisions. The most accessible approaches break the text into individual terms, index the records according to which terms they contain, fit a predictive model of the past decisions, and then create decision inputs from the models.
  • The constrained language used in OPRs makes them amenable to NLP approaches, as shown by the fact that simple models with minimal preprocessing and tuning achieved high levels of accuracy.
  • As compared with state-of-the-art ML approaches (i.e., deep learning), simple linear models based on the presence or absence of key terms achieve similar levels of predictive performance but have the advantage of being inherently interpretable.
  • Key words and phrases that models base predictions on coincide with statements recognizable to expert raters.

Topics

Document Details

Citation

Chicago Manual of Style

Schulker, David, Joshua Williams, Cheryl K. Montemayor, Li Ang Zhang, and Matthew Walsh, The Personnel Records Scoring System: Volume 3, A Methodology for Designing Tools to Support Air Force Human Resources Decisionmaking. Santa Monica, CA: RAND Corporation, 2024. https://www.rand.org/pubs/research_reports/RRA1745-3.html.
BibTeX RIS

Research conducted by

The research reported here was commissioned by the Director of Plans and Integration, Deputy Chief of Staff for Manpower and Personnel, Headquarters U.S. Air Force (AF/A1X) and conducted within the Workforce, Development, and Health Program of RAND Project AIR FORCE.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.