Cover: The Personnel Records Scoring System

The Personnel Records Scoring System

Volume 3, A Methodology for Designing Tools to Support Air Force Human Resources Decisionmaking

Published Feb 27, 2024

by David Schulker, Joshua Williams, Cheryl K. Montemayor, Li Ang Zhang, Matthew Walsh

Download eBook for Free

FormatFile SizeNotes
PDF file 0.8 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. Is it possible for DAF analysts to develop simple models relating key text in officer records to past decisions? How could this be done?
  2. Can NLP approaches be used to create accurate models for OPRs?
  3. What kinds of models are most accurately predictive, and which are most interpretable?
  4. How much overlap is there between the language used by raters and decisionmakers and that used by models to make their predictions?

The Department of the Air Force (DAF) maintains rich records on the knowledge, skills, abilities, and other attributes of its personnel. The personnel system records much of this information as structured lists and free-form text, which creates a need for many processes in which human resources decisionmakers must review the records by hand before issuing a decision. These processes become more difficult with larger populations of officers, as any human judge faces capacity limitations. Thus, there is an opportunity for artificial intelligence applications to improve the quality of inputs for these review processes, helping the human resources management system to become more effective and/or more efficient in meeting DAF strategic goals.

To develop a computational approach to standardize and extract meaning from textual records, the authors reviewed state-of-the-art natural language processing (NLP) and machine learning (ML) approaches. They applied these approaches to text from officer performance reports (OPRs) and used them to predict O-5 and O-6 promotion outcomes and Developmental Education Designation Board scores. The resulting system created for this research is called the Personnel Records Scoring System (PReSS). Aside from the use of ML models to predict board results for future candidates, the authors propose a methodology for using models to generate summary reports that highlight the most-significant statements and detractors contained in a service member's records.

Key Findings

  • DAF analysts can rapidly develop simple models relating key text in officer records to past decisions. The most accessible approaches break the text into individual terms, index the records according to which terms they contain, fit a predictive model of the past decisions, and then create decision inputs from the models.
  • The constrained language used in OPRs makes them amenable to NLP approaches, as shown by the fact that simple models with minimal preprocessing and tuning achieved high levels of accuracy.
  • As compared with state-of-the-art ML approaches (i.e., deep learning), simple linear models based on the presence or absence of key terms achieve similar levels of predictive performance but have the advantage of being inherently interpretable.
  • Key words and phrases that models base predictions on coincide with statements recognizable to expert raters.

Research conducted by

The research reported here was commissioned by the Director of Plans and Integration, Deputy Chief of Staff for Manpower and Personnel, Headquarters U.S. Air Force (AF/A1X) and conducted within the Workforce, Development, and Health Program of RAND Project AIR FORCE.

This report is part of the RAND research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.