Cooperative Multi-Agent Policy Gradients with Sub-optimal Demonstration

2018/12/05 by Peixi Peng, Junliang Xing, Peng, Peixi +1
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Artificial Intelligence (cs.AI) #Computer science #FOS: Computer and information sciences #Optimization and Search Problems #Reinforcement Learning in Robotics #cs.AI

paper · pdf · doi:10.48550/arxiv.1812.01825

openalex publication_date 2018/12/05 · arxiv created 2021/08/19 · arxiv updated 2021/08/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Many reality tasks such as robot coordination can be naturally modelled as multi-agent cooperative system where the rewards are sparse. This paper focuses on learning decentralized policies for such tasks using sub-optimal demonstration. To learn the multi-agent cooperation effectively and tackle the sub-optimality of demonstration, a self-improving learning method is proposed: On the one hand, the centralized state-action values are initialized by the demonstration and updated by the learned decentralized policy to improve the sub-optimality. On the other hand, the Nash Equilibrium are found by the current state-action value and are used as a guide to learn the policy. The proposed method is evaluated on the combat RTS games which requires a high level of multi-agent cooperation. Extensive experimental results on various combat scenarios demonstrate that the proposed method can learn multi-agent cooperation effectively. It significantly outperforms many state-of-the-art demonstration based approaches.

Citations

• Human-level control through deep reinforcement learning
• Mastering the game of Go with deep neural networks and tree search
• Learning Multiagent Communication with Backpropagation
• Generative Adversarial Imitation Learning
• Episodic Exploration for Deep Deterministic Policies: An Application to\n StarCraft Micromanagement Tasks
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games
• Mastering the game of Go without human knowledge
• Deep learning
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Counterfactual Multi-Agent Policy Gradients

Related

• Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework
• Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey
• F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning
• Multi-Agent Actor-Critic with Generative Cooperative Policy Network
• Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning