Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

2020/03/31 by Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt +3 · 8 citations
Computer Science · Mathematics · #cs.LG #cs.MA #stat.ML

paper · pdf

published as Journal of Machine Learning Research 21(178):1-51, 2020 · Extended version of the ICML 2018 conference paper (arXiv:1803.11485)

arxiv created 2020/08/27 · arxiv updated 2020/09/22

Abstract

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.

Citations

• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

Cited by

• Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning
• TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning
• RIDER: 3D RNA Inverse Design with Reinforcement Learning-Guided Diffusion
• Safe and Policy-Compliant Multi-Agent Orchestration for Enterprise AI
• MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory
• Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning
• Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL
• QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?