Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition

2025/05/09 by Dan Qiao, Wenhao Li, Qiao, Dan +7 · 3 citations
Computer Science · #Action selection #Curse of dimensionality #Domain Adaptation and Few-Shot Learning #Explainable Artificial Intelligence (XAI) #Function (biology) #Generative grammar #Generative model #Leverage (statistics) #Online and offline #Regularization (linguistics) #Reinforcement Learning in Robotics #Reinforcement learning

paper · pdf · doi:10.48550/arxiv.2505.05968

published in ArXiv.org

openalex publication_date 2025/05/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts, particularly stemming from the high dimensionality of joint action spaces and the presence of out-of-distribution joint action selections. In this work, we highlight that a fundamental challenge in offline MARL arises from the multi-equilibrium nature of cooperative tasks, which induces a highly multimodal joint behavior policy space coupled with heterogeneous-quality behavior data. This makes it difficult for individual policy regularization to align with a consistent coordination pattern, leading to the policy distribution shift problems. To tackle this challenge, we design a sequential score function decomposition method that distills per-agent regularization signals from the joint behavior policy, which induces coordinated modality selection under decentralized execution constraints. Then we leverage a flexible diffusion-based generative model to learn these score functions from multimodal offline data, and integrate them into joint-action critics to guide policy updates toward high-reward, in-distribution regions under a shared team reward. Our approach achieves state-of-the-art performance across multiple particle environments and Multi-agent MuJoCo benchmarks consistently. To the best of our knowledge, this is the first work to explicitly address the distributional gap between offline and online MARL, paving the way for more generalizable offline policy-based MARL methods.

Citations

• Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
• ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
• Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning
• Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning
• Coordination Failure in Cooperative Offline MARL
• Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation
• Select to Perfect: Imitating desired behavior from large multi-agent data
• MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
• AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation
• Score Regularized Policy Optimization through Diffusion Behavior
• Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
• Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
• Beyond Conservatism: Diffusion Policies in Offline Multi-agent Reinforcement Learning
• Efficient Diffusion Policies for Offline Reinforcement Learning
• MADiff: Offline Multi-agent Learning with Diffusion Models
• A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies
• Order Matters: Agent-by-agent Policy Optimization
• Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning
• CORL: Research-oriented Deep Offline Reinforcement Learning Library
• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
• Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks
• Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification
• Offline Reinforcement Learning with Implicit Q-Learning
• Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning
• Offline Decentralized Multi-Agent Reinforcement Learning
• Model-Based Opponent Modeling
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Score-Based Generative Modeling through Stochastic Differential Equations
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• Game-Theoretic Multiagent Reinforcement Learning
• Denoising Diffusion Probabilistic Models
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• Dota 2 with Large Scale Deep Reinforcement Learning
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
• Behavior Regularized Offline Reinforcement Learning
• Generative Modeling by Estimating Gradients of the Data Distribution
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Learning with Opponent-Learning Awareness
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Masked Autoregressive Flow for Density Estimation
• A Kernelized Stein Discrepancy for Goodness-of-fit Tests and Model Evaluation
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• The Complexity of Decentralized Control of Markov Decision Processes
• The Complexity of Decentralized Control of Markov Decision Processes

Cited by

• Multi-agent Coordination via Flow Matching
• OM2P: Offline Multi-Agent Mean-Flow Policy
• Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL

Related

• ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
• Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
• Coordination Failure in Cooperative Offline MARL
• Offline Decentralized Multi-Agent Reinforcement Learning
• Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning