MMD-MIX: Value Function Factorisation with Maximum Mean Discrepancy for Cooperative Multi-Agent Reinforcement Learning

2021/06/22 by Zhiwei Xu, Xu, Zhiwei, Dapeng Li +5 · 1 citation
Computer Science · Engineering · Mathematics · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Artificial intelligence #Autonomous Vehicle Technology and Safety #Bellman equation #Class (philosophy) #Computer science #Decomposition #FOS: Computer and information sciences #Function (biology) #Machine Learning (cs.LG) #Machine learning #Mathematical optimization #Mathematics #Multiagent Systems (cs.MA) #Randomness #Reinforcement Learning in Robotics #Reinforcement learning #Statistics #Value (mathematics) #cs.AI #cs.LG #cs.MA

paper · pdf · doi:10.48550/arxiv.2106.11652

published in arXiv (Cornell University) (Cornell University) · 7 pages, 2 figures, 2 tables. Accepted by IJCNN 2021

arxiv created 2021/06/22 · openalex publication_date 2021/06/22 · arxiv updated 2021/06/23 · openalex created_date 2021/07/05 · openalex updated_date 2026/08/05

Abstract

In the real world, many tasks require multiple agents to cooperate with each other under the condition of local observations. To solve such problems, many multi-agent reinforcement learning methods based on Centralized Training with Decentralized Execution have been proposed. One representative class of work is value decomposition, which decomposes the global joint Q-value Q_jt into individual Q-values Q_a to guide individuals' behaviors, e.g. VDN (Value-Decomposition Networks) and QMIX. However, these baselines often ignore the randomness in the situation. We propose MMD-MIX, a method that combines distributional reinforcement learning and value decomposition to alleviate the above weaknesses. Besides, to improve data sampling efficiency, we were inspired by REM (Random Ensemble Mixture) which is a robust RL algorithm to explicitly introduce randomness into the MMD-MIX. The experiments demonstrate that MMD-MIX outperforms prior baselines in the StarCraft Multi-Agent Challenge (SMAC) environment.

Citations

• Playing Atari with Deep Reinforcement Learning
• Auto-Encoding Variational Bayes
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• MMD GAN: Towards Deeper Understanding of Moment Matching Network
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Learning to Schedule Communication in Multi-agent Reinforcement Learning
• Counterfactual Multi-Agent Policy Gradients
• Distributional Reinforcement Learning with Quantile Regression
• Solving Rubik's Cube with a Robot Hand

Related

• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• MCMARL: Parameterizing Value Function via Mixture of Categorical Distributions for Multi-Agent Reinforcement Learning
• Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning
• A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning