Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration

2025/05/08 by Andreas Kontogiannis, Konstantinos Papathanasiou, Kontogiannis, Andreas +9
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2505.05262

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Learning to cooperate in distributed partially observable environments with no communication abilities poses significant challenges for multi-agent deep reinforcement learning (MARL). This paper addresses key concerns in this domain, focusing on inferring state representations from individual agent observations and leveraging these representations to enhance agents' exploration and collaborative task execution policies. To this end, we propose a novel state modelling framework for cooperative MARL, where agents infer meaningful belief representations of the non-observable state, with respect to optimizing their own policies, while filtering redundant and less informative joint state information. Building upon this framework, we propose the MARL SMPE algorithm. In SMPE, agents enhance their own policy's discriminative abilities under partial observability, explicitly by incorporating their beliefs into the policy network, and implicitly by adopting an adversarial type of exploration policies which encourages agents to discover novel, high-value states while improving the discriminative abilities of others. Experimentally, we show that SMPE outperforms state-of-the-art MARL algorithms in complex fully cooperative tasks from the MPE, LBF, and RWARE benchmarks.

Citations

• An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
• Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learning
• Fast Peer Adaptation with Context-aware Exploration
• Boosting Value Decomposition via Unit-Wise Attentive State Representation for Cooperative Multi-Agent Reinforcement Learning
• Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning
• Decision-making with Speculative Opponent Models
• Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning
• Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
• Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic
• Learning Intuitive Policies Using Action Features
• GCS: Graph-based Coordination Strategy for Multi-Agent Reinforcement Learning
• Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration
• Celebrating Diversity in Shared Multi-Agent Reinforcement Learning
• SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Off-Belief Learning
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• Deep Interactive Bayesian Reinforcement Learning via Meta-Learning
• Theory of Mind with Guilt Aversion Facilitates Cooperative Reinforcement Learning
• Value-Decomposition Multi-Agent Actor-Critics
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks
• The Emergence of Individuality
• Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
• Variational Autoencoders for Opponent Modeling in Multi-Agent Systems
• Agent Modeling as Auxiliary Task for Deep Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Modeling Others using Oneself in Multi-Agent Reinforcement Learning
• Learning with Opponent-Learning Awareness
• Attention Is All You Need
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• Reasoning about Hypothetical Agent Behaviours and their Parameters
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Opponent Modeling in Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning

Related

• Information State Embedding in Partially Observable Cooperative Multi-Agent Reinforcement Learning
• Belief States for Cooperative Multi-Agent Reinforcement Learning under Partial Observability
• A Review of Cooperative Multi-Agent Deep Reinforcement Learning
• Policy Optimization in Multi-Agent Settings under Partially Observable Environments
• S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?