Major-Minor Mean Field Multi-Agent Reinforcement Learning

2023/03/19 by Kai Cui, Cui, Kai, Christian Fabián +3 · 2 citations
Decision Sciences · Computer Science · Economics, Econometrics and Finance · #Auction Theory and Applications #Reinforcement Learning in Robotics #Climate Change Policy and Economics

paper · pdf · doi:10.48550/arxiv.2303.10665

Abstract

Multi-agent reinforcement learning (MARL) remains difficult to scale to many agents. Recent MARL using Mean Field Control (MFC) provides a tractable and rigorous approach to otherwise difficult cooperative MARL. However, the strict MFC assumption of many independent, weakly-interacting agents is too inflexible in practice. We generalize MFC to instead simultaneously model many similar and few complex agents -- as Major-Minor Mean Field Control (M3FC). Theoretically, we give approximation results for finite agent control, and verify the sufficiency of stationary policies for optimality together with a dynamic programming principle. Algorithmically, we propose Major-Minor Mean Field MARL (M3FMARL) for finite agent systems instead of the limiting system. The algorithm is shown to approximate the policy gradient of the underlying M3FC MDP. Finally, we demonstrate its capabilities experimentally in various scenarios. We observe a strong performance in comparison to state-of-the-art policy gradient MARL methods.

Cited by

• An Overview of Some Extensions of Mean Field Games beyond Perfect Homogeneity and Anonymity
• Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning

Related

• Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)
• Mean Field Multi-Agent Reinforcement Learning
• Multi Type Mean Field Reinforcement Learning
• Discrete-Time Mean Field Control with Environment States
• Graphon Mean-Field Control for Cooperative Multi-Agent Reinforcement Learning