Q-Learning in Regularized Mean-field Games

2020/03/24 by Berkay Anahtarci, Berkay Anahtarcı, Anahtarci, Berkay +6 · 6 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · Mathematics · #FOS: Computer and information sciences #FOS: Electrical engineering #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC) #Receptor Mechanisms and Signaling #Systems and Control (eess.SY) #cs.LG #cs.SY #eess.SY #electronic engineering #information engineering #math.OC

paper · pdf · doi:10.48550/arxiv.2003.12151

32 pages. arXiv admin note: text overlap with arXiv:1912.13309

openalex publication_date 2020/03/24 · arxiv created 2022/11/10 · arxiv updated 2022/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. Regularization is introduced by adding a strongly concave regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. The regularization term in general makes reinforcement learning algorithm more robust to the system components. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

Citations

• Learning in Discounted-cost and Average-cost Mean-field Games

Cited by

• Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces
• Entropy Regularization in Mean-Field Games of Optimal Stopping
• Learning in Stackelberg Markov Games
• Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning
• Approximation of Discrete-Time Infinite-Horizon Mean-Field Equilibria via Finite-Horizon Mean-Field Equilibria
• Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games
• Neural Mean-Field Games: Extending Mean-Field Game Theory with Neural Stochastic Differential Equations

Related

• Unified Reinforcement Q-Learning for Mean Field Game and Control Problems
• Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems
• Reinforcement Learning for Mean Field Games, with Applications to Economics
• Learning in games via reinforcement and regularization
• Learning Mean-Field Games