QTRAN: Learning to Factorize with Transformation for Cooperative\n Multi-Agent Reinforcement Learning

2019/05/14 by Kyunghwan Son, Son, Kyunghwan, Dae Woo Kim +7 · 61 citations
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics #Robotic Path Planning Algorithms

paper · pdf · doi:10.48550/arxiv.1905.05408

openalex publication_date 2019/05/14 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

We explore value-based solutions for multi-agent reinforcement learning\n(MARL) tasks in the centralized training with decentralized execution (CTDE)\nregime popularized recently. However, VDN and QMIX are representative examples\nthat use the idea of factorization of the joint action-value function into\nindividual ones for decentralized execution. VDN and QMIX address only a\nfraction of factorizable MARL tasks due to their structural constraint in\nfactorization such as additivity and monotonicity. In this paper, we propose a\nnew factorization method for MARL, QTRAN, which is free from such structural\nconstraints and takes on a new approach to transforming the original joint\naction-value function into an easily factorizable one, with the same optimal\nactions. QTRAN guarantees more general factorization than VDN or QMIX, thus\ncovering a much wider class of MARL tasks than does previous methods. Our\nexperiments for the tasks of multi-domain Gaussian-squeeze and modified\npredator-prey demonstrate QTRAN's superior performance with especially larger\nmargins in games whose payoffs penalize non-cooperative behavior more\naggressively.\n

Cited by

• ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modeling
• Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus
• StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm
• Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition
• MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning
• Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition
• Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning
• Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning
• A Negotiation-Based Multi-Agent Reinforcement Learning Approach for Dynamic Scheduling of Reconfigurable Manufacturing Systems
• Partial Action Replacement: Tackling Distribution Shift in Offline MARL
• MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning
• Multi-agent Coordination via Flow Matching
• Multi-party Agent Relation Sampling for Multi-party Ad Hoc Teamwork
• Collaborative Scheduling of Time-dependent UAVs,Vehicles and Workers for Crowdsensing in Disaster Response
• Learning Nearly Decomposable Value Functions Via Communication Minimization
• Sequential Multi-Agent Dynamic Algorithm Configuration
• Mimicking Evolution with Reinforcement Learning
• IFS: Information Flow Structure for Multi-agent Ad Hoc System
• High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning
• Value-Decomposition Multi-Agent Actor-Critics
• AOAD-MAT: Transformer-based multi-agent deep reinforcement learning model considering agents' order of action decisions
• Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
• AI Agents for the Dhumbal Card Game: A Comparative Study
• Regularized Softmax Deep Multi-Agent Q-Learning
• Expandable Decision-Making States for Multi-Agent Deep Reinforcement Learning in Soccer Tactical Analysis
• Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
• A Framework for Scalable Heterogeneous Multi-Agent Adversarial Reinforcement Learning in IsaacLab
• Triple-BERT: Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms?
• Embodied AI: From LLMs to World Models
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems
• Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problem
• HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
• K-Level Policy Gradients for Multi-Agent Reinforcement Learning
• Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Offline Decentralized Multi-Agent Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement\n Learning
• MACTAS: Self-Attention-Based Module for Inter-Agent Communication in Multi-Agent Reinforcement Learning
• Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach
• Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Policy Optimization in Multi-Agent Settings under Partially Observable Environments
• Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
• Concept Learning for Cooperative Multi-Agent Reinforcement Learning
• Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints
• Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise
• Multi-Year Maintenance Planning for Large-Scale Infrastructure Systems: A Novel Network Deep Q-Learning Approach
• SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning
• Multi-Agent Guided Policy Optimization
• Learning to Coordinate via Multiple Graph Neural Networks
• Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• ToMacVF : Temporal Macro-action Value Factorization for Asynchronous Multi-Agent Reinforcement Learning
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• From General Relation Patterns to Task-Specific Decision-Making in Continual Multi-Agent Coordination
• Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
• Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning
• RALLY: Role-Adaptive LLM-Driven Yoked Navigation for Agentic UAV Swarms
• Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning

Related

• Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning