Examining Policy Entropy of Reinforcement Learning Agents for Personalization Tasks

2022/11/21 by Anton Dereventsov, Dereventsov, Anton, Andrew Starnes +3
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA) #Optimization and Control (math.OC) #Reinforcement Learning in Robotics #Smart Grid Energy Management

paper · pdf · doi:10.48550/arxiv.2211.11869

openalex publication_date 2022/11/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This effort is focused on examining the behavior of reinforcement learning systems in personalization environments and detailing the differences in policy entropy associated with the type of learning algorithm utilized. We demonstrate that Policy Optimization agents often possess low-entropy policies during training, which in practice results in agents prioritizing certain actions and avoiding others. Conversely, we also show that Q-Learning agents are far less susceptible to such behavior and generally maintain high-entropy policies throughout training, which is often preferable in real-world applications. We provide a wide range of numerical experiments as well as theoretical justification to show that these differences in entropy are due to the type of learning being employed.

Related

• Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks
• Robust Policy Optimization in Deep Reinforcement Learning
• Understanding the impact of entropy on policy optimization
• Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning
• Policy Optimization Reinforcement Learning with Entropy Regularization