Cover: Modeling Agent Behaviors for Policy Analysis Via Reinforcement Learning

Modeling Agent Behaviors for Policy Analysis Via Reinforcement Learning

Published in: 2020 19th IEEE International Conference on Machine Learning and Applications (2021). doi: 10.1109/ICMLA51294.2020.00043

Posted on Mar 9, 2021

by Osonde A. Osoba, Raffaele Vardavas, Justin Grana, Rushil Zutshi, Amber Jaycocks

Agent-based Models (ABMs) are valuable tools for policy analysis. ABMs help analysts explore the emergent consequences of regulatory and policy interventions in multi-agent decision-making settings. But the validity of inferences drawn from ABM explorations depends on the quality of the ABM agents' behavioral models. Prior approaches for specifying behaviors have limitations. This paper examines the value of reinforcement learning (RL) models as adaptive, high-performing, and behaviorally-valid models of agent decision-making in ABMs. We discuss the value of RL for modeling agents' utility-maximizing behaviors in policy-relevant ABMs. We address the problem of adapting RL algorithms to handle multi-agency in games by adapting and extending methods from recent literature. We evaluate examples of such RL-based ABM agents via experiments on two policy-relevant ABMs: a Minority Game ABM, and an ABM of Influenza Transmission. The RL behavioral models can outperform the default adaptive behavioral models. We also run analytic experiments on our RL-equipped ABMs: explorations of the effects of dynamic behavioral heterogeneity in a population, the impact of social network factors on adaptability, and the emergence of synchronization in a community. Our results suggest that the RL formalism can be an efficient abstraction for behavioral models in ABMs.

This report is part of the RAND external publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.