2022/09/17 by Kefan Su, Su, Kefan, Siyuan Zhou +8 · 1 citation
Computer Science · Decision Sciences · #Distributed Control Multi-Agent Systems #FOS: Computer and information sciences #Game Theory and Applications #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.2209.08244
openalex publication_date 2022/09/17 · openalex created_date 2022/09/21 · openalex updated_date 2026/07/28
Decentralized learning has shown great promise for cooperative multi-agent reinforcement learning (MARL). However, non-stationarity remains a significant challenge in fully decentralized learning. In the paper, we tackle the non-stationarity problem in the simplest and fundamental way and propose multi-agent alternate Q-learning (MA2QL), where agents take turns updating their Q-functions by Q-learning. MA2QL is a minimalist approach to fully decentralized cooperative MARL but is theoretically grounded. We prove that when each agent guarantees ε-convergence at each turn, their joint policy converges to a Nash equilibrium. In practice, MA2QL only requires minimal changes to independent Q-learning (IQL). We empirically evaluate MA2QL on a variety of cooperative multi-agent tasks. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.