QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

2019/05/14 by Kyunghwan Son, Daewoo Kim, Dae Woo Kim +8 · 99 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms #cs.AI #cs.LG #cs.MA #stat.ML

paper · pdf · doi:10.48550/arxiv.1905.05408

18 pages; Accepted to ICML 2019

arxiv created 2019/05/14 · openalex publication_date 2019/05/14 · arxiv updated 2019/05/15 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones for decentralized execution. VDN and QMIX address only a fraction of factorizable MARL tasks due to their structural constraint in factorization such as additivity and monotonicity. In this paper, we propose a new factorization method for MARL, QTRAN, which is free from such structural constraints and takes on a new approach to transforming the original joint action-value function into an easily factorizable one, with the same optimal actions. QTRAN guarantees more general factorization than VDN or QMIX, thus covering a much wider class of MARL tasks than does previous methods. Our experiments for the tasks of multi-domain Gaussian-squeeze and modified predator-prey demonstrate QTRAN's superior performance with especially larger margins in games whose payoffs penalize non-cooperative behavior more aggressively.

Cited by

• ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modeling
• Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
• StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm
• Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
• MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning
• Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition
• Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning
• Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning
• A Negotiation-Based Multi-Agent Reinforcement Learning Approach for Dynamic Scheduling of Reconfigurable Manufacturing Systems
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL
• MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning
• Multi-agent Coordination via Flow Matching
• Multi-party Agent Relation Sampling for Multi-party Ad Hoc Teamwork
• Collaborative Scheduling of Time-dependent UAVs,Vehicles and Workers for Crowdsensing in Disaster Response
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Sequential Multi-Agent Dynamic Algorithm Configuration
• Mimicking Evolution with Reinforcement Learning
• IFS: Information Flow Structure for Multi-agent Ad Hoc System
• High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning
• Value-Decomposition Multi-Agent Actor-Critics
• AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisions
• Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
• AI Agents for the Dhumbal Card Game: A Comparative Study
• Regularized Softmax Deep Multi-Agent Q-Learning
• Expandable Decision-Making States for Multi-Agent Deep Reinforcement Learning in Soccer Tactical Analysis
• Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
• A Framework for Scalable Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLab
• Triple-BERT: Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms?
• Embodied AI: From LLMs to World Models
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems
• Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problem
• HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
• K-Level Policy Gradients for Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Offline Decentralized Multi-Agent Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning
• MACTAS: Self-Attention-Based Inter-Agent Communication in Multi-Agent Reinforcement Learning with Action-Value Function Decomposition
• Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach
• Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Policy Optimization in Multi-Agent Settings under Partially Observable Environments
• Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
• Concept Learning for Cooperative Multi-Agent Reinforcement Learning
• Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints
• Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise
• Multi-Year Maintenance Planning for Large-Scale Infrastructure Systems: A Novel Network Deep Q-Learning Approach
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Multi-Agent Guided Policy Optimization
• Learning to Coordinate via Multiple Graph Neural Networks
• Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination
• Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
• Settling the Variance of Multi-Agent Policy Gradients
• Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning
• RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarms
• Curriculum Learning With Counterfactual Group Relative Policy Advantage For Multi-Agent Reinforcement Learning
• Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning
• Semi-On-Policy Training for Sample Efficient Multi-Agent Policy Gradients
• Sequence Modeling for N-Agent Ad Hoc Teamwork
• Multi-Agent Intention Sharing via Leader-Follower Forest
• Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core
• Multi-agent Markov Entanglement
• Action Dependency Graphs for Globally Optimal Coordinated Reinforcement Learning
• Calculus of Consent via MARL: Legitimating the Collaborative Governance\n Supplying Public Goods
• V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
• DFAC Framework: Factorizing the Value Function via Quantile Mixture for\n Multi-Agent Distributional Q-Learning
• Adaptive Episode Length Adjustment for Multi-agent Reinforcement Learning
• A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
• MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations
• Divergence-Regularized Multi-Agent Actor-Critic
• CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
• SyncPlan: Long-Horizon LLM Coordination with Explicit Synchronization and Adaptive Correction
• Dynamic Sight Range Selection in Multi-Agent Reinforcement Learning
• ROMA: Multi-Agent Reinforcement Learning with Emergent Roles
• Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors
• Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning
• Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning
• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• HYGMA: Hypergraph Coordination Networks with Dynamic Grouping for Multi-Agent Reinforcement Learning
• A Multi-Agent Reinforcement Learning Approach for Cooperative Air-Ground-Human Crowdsensing in Emergency Rescue
• Bi-level Mean Field: Dynamic Grouping for Large-Scale MARL
• Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
• RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents
• TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning
• Investigating Adaptive Tuning of Assistive Exoskeletons Using Offline Reinforcement Learning: Challenges and Insights
• Robust Multi-agent Communication Based on Decentralization-Oriented Adversarial Training
• Q-function Decomposition with Intervention Semantics with Factored Action Spaces
• Exploiting inter-agent coupling information for efficient reinforcement learning of cooperative LQR
• Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning
• QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?
• Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability

Related

• Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning