If MaxEnt RL is the Answer, What is the Question?

2019/10/04 by Benjamin Eysenbach, Sergey Levine, Eysenbach, Benjamin +1 · 10 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · Neuroscience · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural dynamics and brain function #Receptor Mechanisms and Signaling #Reinforcement Learning in Robotics #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1910.01913

arxiv created 2019/10/04 · openalex publication_date 2019/10/04 · arxiv updated 2019/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.

Citations

• Artificial intelligence: a modern approach
• Continuous control with deep reinforcement learning
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Robust Adversarial Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Maximum a Posteriori Policy Optimisation
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Variational Bayesian Reinforcement Learning with Regret Bounds
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Adversarial Examples Are Not Bugs, They Are Features
• Efficient Exploration via State Marginal Matching

Cited by

• StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm
• Off-policy Reinforcement Learning with Model-based Exploration Augmentation
• Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
• State Entropy Regularization for Robust Reinforcement Learning
• Diversity in Action: General-Sum Multi-Agent Continuous Inverse Optimal Control
• Soft Hindsight Experience Replay
• Divergence-Regularized Multi-Agent Actor-Critic
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems

Related

• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• When Maximum Entropy Misleads Policy Optimization
• Failure Modes of Maximum Entropy RLHF
• A Regularized Opponent Model with Maximum Entropy Objective
• Optimal Policies Tend to Seek Power