QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

2018/03/30 by Tabish Rashid, Rashid, Tabish, Mikayel Samvelyan +9 · 51 citations
Computer Science · Decision Sciences · #Auction Theory and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mobile Crowdsensing and Crowdsourcing #Multiagent Systems (cs.MA) #Open Source Software Innovations #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1803.11485

openalex publication_date 2018/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

In many real-world settings, a team of agents must coordinate their behaviour while acting in a decentralised way. At the same time, it is often possible to train the agents in a centralised fashion in a simulated or laboratory setting, where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a network that estimates joint action-values as a complex non-linear combination of per-agent values that condition only on local observations. We structurally enforce that the joint-action value is monotonic in the per-agent values, which allows tractable maximisation of the joint action-value in off-policy learning, and guarantees consistency between the centralised and decentralised policies. We evaluate QMIX on a challenging set of StarCraft II micromanagement tasks, and show that QMIX significantly outperforms existing value-based multi-agent reinforcement learning methods.

Cited by

• Intrinsic-Motivation Multi-Robot Social Formation Navigation with Coordinated Exploration
• IPPO Learns the Game, Not the Team: A Study on Generalization in Heterogeneous Agent Teams
• HiveMind: Contribution-Guided Online Prompt Optimization of LLM Multi-Agent Systems
• Dynamic Configuration of On-Street Parking Spaces using Multi Agent Reinforcement Learning
• Extending NGU to Multi-Agent RL: A Preliminary Study
• Emergent Coordination and Phase Structure in Independent Multi-Agent Reinforcement Learning
• MAESTRO: Multi-Agent Environment Shaping through Task and Reward Optimization
• Adversarial Attack on Black-Box Multi-Agent by Adaptive Perturbation
• Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition
• Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning
• Who Gets the Reward, Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
• A Negotiation-Based Multi-Agent Reinforcement Learning Approach for Dynamic Scheduling of Reconfigurable Manufacturing Systems
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL
• Network-Constrained Policy Optimization for Adaptive Multi-agent Vehicle Routing
• Multi-party Agent Relation Sampling for Multi-party Ad Hoc Teamwork
• Ripple Effect Protocol: Coordinating Agent Populations
• Human Machine Social Hybrid Intelligence:A Collaborative Decision Making Framework for Large Model Agent Groups and Human Experts
• On the Fundamental Limitations of Decentralized Learnable Reward Shaping in Cooperative Multi-Agent Reinforcement Learning
• IFS: Information Flow Structure for Multi-agent Ad Hoc System
• High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning
• Socialized Learning and Emergent Behaviors in Multi-Agent Systems based on Multimodal Large Language Models
• STEMS: Spatial-Temporal Enhanced Safe Multi-Agent Coordination for Building Energy Management
• Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
• Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning
• Coordinated Strategies in Realistic Air Combat by Hierarchical Multi-Agent Reinforcement Learning
• A Joint Learning Approach to Hardware Caching and Prefetching
• Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
• MAT-Agent: Adaptive Multi-Agent Training Optimization
• LLMs as Policy-Agnostic Teammates: A Case Study in Human Proxy Design for Heterogeneous Agent Teams
• MARLIN: Multi-Agent Reinforcement Learning with Murmuration Intelligence and LLM Guidance for Reservoir Management
• Learning to Lead Themselves: Agentic AI in MAS using MARL
• The Heterogeneous Multi-Agent Challenge
• AI Agent Access (A\³) Network: An Embodied, Communication-Aware Multi-Agent Framework for 6G Coverage
• Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning
• Fine-Grained AI Model Caching and Downloading With Coordinated Multipoint Broadcasting in Multi-Cell Edge Networks
• Co-Alignment: Rethinking Alignment as Bidirectional Human-AI Cognitive Adaptation
• Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
• Scalable Solution Methods for Dec-POMDPs with Deterministic Dynamics
• Skill-Aligned Fairness in Multi-Agent Learning for Collaboration in Healthcare
• An Improved Multi-Agent Algorithm for Cooperative and Competitive Environments by Identifying and Encouraging Cooperation among Agents
• MACTAS: Self-Attention-Based Module for Inter-Agent Communication in Multi-Agent Reinforcement Learning
• Centralized Permutation Equivariant Policy for Cooperative Multi-Agent Reinforcement Learning
• Constrained Black-Box Attacks Against Multi-Agent Reinforcement Learning
• Fault Tolerant Multi-Agent Learning with Adversarial Budget Constraints
• Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety
• Concept Learning for Cooperative Multi-Agent Reinforcement Learning
• Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise
• Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
• DPBench: Structural Determinants of Multi-Agent LLM Coordination Under Simultaneous Resource Contention
• A Learning Framework For Cooperative Collision Avoidance of UAV Swarms Leveraging Domain Knowledge

Related

• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative\n Multi-Agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning
• Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning