- How could an AI-enabled performance-scoring system that mimics the process by which human experts judge performance narratives help senior leaders take full advantage of performance records when making talent management decisions?
- What are the potential consequences and implications of implementing such an AI system?
Both private and public organizations are increasingly taking advantage of improvements in computing power, data availability, and analytic capabilities to improve business processes. These trends have prompted U.S. Department of Defense policymakers to become more interested in whether adopting data-enabled methods would facilitate more-effective management of department personnel. In this report, RAND researchers explore one such application that would enable the U.S. Air Force to leverage existing data for improved human resource management (HRM) policies and practices. Specifically, the researchers develop a performance-scoring system that uses artificial intelligence (AI) and machine learning, which would enable the expanded use of performance narratives in HRM processes. The main purpose of this report is to serve as a worked example (i.e., a step-by-step solution to a problem) for Air Force policymakers as they consider how to approach the potential ways in which AI can improve HRM processes.
- The researchers apply the cross-industry standard process for data-mining to the AI application explored in this analysis. This task requires two primary data inputs: (1) a sufficiently large sample of officer performance narratives and (2) outcome labels that provide information about which narratives indicate the best job performance.
- The researchers find that there is a business need for the AI performance-scoring system. The system could be used as a tool to facilitate policy analysis, assist development teams, enable professional development, or aid in competitive selection decisions.
- Extracting and digitizing large amounts of officer evaluation data from the existing archive is feasible, but, because the process of digitizing the text from older documents requires extensive tuning and computation time, the process could present a potential challenge to future efforts by Air Force practitioners.
- Initial model results are promising: Standard machine-learning algorithms accurately predicted an officer's performance quality by identifying known signals in the performance narrative text without explicit programming.
- Implementation concerns regarding privacy, fairness, explainability, and other unintended consequences are greatest if the system were used to make decisions that would alter officers' careers. These considerations could be less of a barrier to implementing the system for other purposes.
- Policymakers should position systems and policies to take advantage of analytic uses of data that HRM processes generate.
- Natural language processing techniques can reduce the need to pre-quantify information at the data collection stage.
- Human resources managers should consider AI applications to be an enabler of better policies rather than a substitute for human decisionmaking.
Table of Contents
The Business Need for Performance Metrics in Human Resource Management Decisions
Understanding Officer Evaluation System Data
Processing Performance Narratives to Create Analytic Data
Modeling the Relationship Between Performance Narratives and Promotion
Evaluating Other Implementation Considerations
Conclusion and Policy Implications