ROMA: Multi-Agent Reinforcement Learning with Emergent Roles

2020/03/18 by Tonghan Wang, Heng Dong, Wang, Tonghan +5 · 32 citations
Computer Science · Mathematics · Psychology · #Adaptability #Adaptation (eye) #Artificial intelligence #Benchmark (surveying) #Benchmarking #Business #Cognitive science #Computer science #Construct (python library) #Domain (mathematical analysis) #Embedding #FOS: Computer and information sciences #Flexibility (engineering) #Mathematics #Multi-Agent Systems and Negotiation #Multiagent Systems (cs.MA) #Multimodal Machine Learning Applications #Programming language #Psychology #Reinforcement Learning in Robotics #Reinforcement learning #cs.MA

paper · pdf · doi:10.48550/arxiv.2003.08039

published in arXiv (Cornell University) (Cornell University) · Thirty-seventh International Conference on Machine Learning (ICML 2020)

openalex publication_date 2020/03/18 · arxiv created 2020/07/04 · arxiv updated 2020/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The role concept provides a useful tool to design and understand complex multi-agent systems, which allows agents with a similar role to share similar behaviors. However, existing role-based methods use prior domain knowledge and predefine role structures and behaviors. In contrast, multi-agent reinforcement learning (MARL) provides flexibility and adaptability, but less efficiency in complex tasks. In this paper, we synergize these two paradigms and propose a role-oriented MARL framework (ROMA). In this framework, roles are emergent, and agents with similar roles tend to share their learning and to be specialized on certain sub-tasks. To this end, we construct a stochastic role embedding space by introducing two novel regularizers and conditioning individual policies on roles. Experiments show that our method can learn specialized, dynamic, and identifiable roles, which help our method push forward the state of the art on the StarCraft II micromanagement benchmark. Demonstrative videos are available at https://sites.google.com/view/romarl/.

Citations

• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to\n StarCraft Micromanagement Tasks
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• StarCraft II: A New Challenge for Reinforcement Learning
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Counterfactual Multi-Agent Policy Gradients
• Emergence of Grounded Compositional Language in Multi-Agent Populations
• Learning Fairness in Multi-Agent Systems

Cited by

• Rethinking Multi-Agent Intelligence Through the Lens of Small-World Networks
• StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm
• Agent-Kernel: A MicroKernel Multi-Agent System Framework for Adaptive Social Simulation Powered by LLMs
• Adaptively Coordinating with Novel Partners via Learned Latent Strategies
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
• IFS: Information Flow Structure for Multi-agent Ad Hoc System
• A Principle of Targeted Intervention for Multi-Agent Reinforcement Learning
• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
• Voting-Bloc Entropy: A New Metric for DAO Decentralization
• Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learning
• Local-Canonicalization Equivariant Graph Neural Networks for Sample-Efficient and Generalizable Swarm Robot Control
• HLSMAC: A New StarCraft Multi-Agent Challenge for High-Level Strategic Decision-Making
• When Is Diversity Rewarded in Cooperative Multi-Agent Learning?
• Skill-Aligned Fairness in Multi-Agent Learning for Collaboration in Healthcare
• A Taxonomy of Hierarchical Multi-Agent Systems: Design Patterns, Coordination Mechanisms, and Industrial Applications
• Traffic Load-Aware Resource Management Strategy for Underwater Wireless Sensor Networks
• Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertise
• Dual-level Behavioral Consistency for Inter-group and Intra-group Coordination in Multi-Agent Systems
• A MARL-based Approach for Easing MAS Organization Engineering
• R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning
• Adaptive Episode Length Adjustment for Multi-agent Reinforcement Learning
• URB -- Urban Routing Benchmark for RL-equipped Connected Autonomous Vehicles
• Dynamic Sight Range Selection in Multi-Agent Reinforcement Learning
• Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning
• HYGMA: Hypergraph Coordination Networks with Dynamic Grouping for Multi-Agent Reinforcement Learning
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning

Related

• RODE: Learning Roles to Decompose Multi-Agent Tasks
• Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles
• Learning to Transfer Role Assignment Across Team Sizes
• R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning
• Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions