Reinforcement Learning for Discrete-time LQG Mean Field Social Control Problems with Unknown Dynamics

2025/07/02 by Zhang, Hanfang, Bing‐Chang Wang, Shuo Chen +2
Computer Science · #Adaptive Dynamic Programming Control #Distributed Control Multi-Agent Systems #FOS: Mathematics #Optimization and Control (math.OC) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2507.01420

openalex publication_date 2025/07/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper studies the discrete-time linear-quadratic-Gaussian mean field (MF) social control problem in an infinite horizon, where the dynamics of all agents are unknown. The objective is to design a reinforcement learning (RL) algorithm to approximate the decentralized asymptotic optimal social control in terms of two algebraic Riccati equations (AREs). In this problem, a coupling term is introduced into the system dynamics to capture the interactions among agents. This causes the equivalence between model-based and model-free methods to be invalid, which makes it difficult to directly apply traditional model-free algorithms. Firstly, under the assumptions of system stabilizability and detectability, a model-based policy iteration algorithm is proposed to approximate the stabilizing solution of the AREs. The algorithm is proven to be convergent in both cases of semi-positive definite and indefinite weight matrices. Subsequently, by adopting the method of system transformation, a model-free RL algorithm is designed to solve for asymptotic optimal social control. During the iteration process, the updates are performed using data collected from any two agents and MF state. Finally, a numerical case is provided to verify the effectiveness of the proposed algorithm.

Citations

• Mean Field LQG Social Optimization: A Reinforcement Learning Approach
• Mean Field Game Approach to Non-Pharmaceutical Interventions in a Social Structure model of Epidemics
• Scalable Deep Reinforcement Learning Algorithms for Mean Field Games
• Approximately Solving Mean Field Games via Entropy-Regularized Deep\n Reinforcement Learning
• LQG Mean Field Games with a Major Agent: Nash Certainty Equivalence versus Probabilistic Approach
• Stationary fully nonlinear mean-field games
• Unified Reinforcement Q-Learning for Mean Field Game and Control Problems
• Robust Mean Field Social Control Problems with Applications in Analysis of Opinion Dynamics
• Mean Field Linear Quadratic Control: Uniform Stabilization and Social Optimality
• Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis
• Linear quadratic mean field games: Asymptotic solvability and relation to the fixed point approach
• Semi-Explicit Solutions to some Non-Linear Non-Quadratic Mean-Field-Type Games: A Direct Method
• On the convergence of closed-loop Nash equilibria to the mean field game limit
• Dynamic Collective Choice: Social Optima
• The master equation and the convergence problem in mean field games
• On the connection between symmetric N-player games and mean field games

Related

• Mean Field LQG Social Optimization: A Reinforcement Learning Approach
• Robust Mean Field Social Control: A Unified Reinforcement Learning Framework
• Social Optima in Robust Mean Field LQG Control: From Finite to Infinite Horizon
• Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problems
• Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods