On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation

2023/05/18 by Jiawei Huang, Huang, Jiawei, Batuhan Yardim +3 · 1 citation
Decision Sciences · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Innovation Diffusion and Forecasting #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2305.11283

openalex publication_date 2023/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

In this paper, we study the fundamental statistical efficiency of Reinforcement Learning in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general model-based function approximation. We introduce a new concept called Mean-Field Model-Based Eluder Dimension (MF-MBED), which characterizes the inherent complexity of mean-field model classes. We show that a rich family of Mean-Field RL problems exhibits low MF-MBED. Additionally, we propose algorithms based on maximal likelihood estimation, which can return an ε-optimal policy for MFC or an ε-Nash Equilibrium policy for MFG. The overall sample complexity depends only polynomially on MF-MBED, which is potentially much lower than the size of state-action space. Compared with previous works, our results only require the minimal assumptions including realizability and Lipschitz continuity.

Cited by

• Approximation of Discrete-Time Infinite-Horizon Mean-Field Equilibria via Finite-Horizon Mean-Field Equilibria

Related

• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL
• When is Mean-Field Reinforcement Learning Tractable and Relevant?
• Unified Reinforcement Q-Learning for Mean Field Game and Control Problems
• Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems
• Reinforcement Learning for Mean Field Games, with Applications to Economics