Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

2021/12/22 by Rui Zhao, Jinming Song, Zhao, Rui +14 · 21 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Reinforcement Learning in Robotics #cs.AI

paper · pdf · doi:10.48550/arxiv.2112.11701

Accepted by NeurIPS Cooperative AI Workshop, 2021, link: https://www.cooperativeai.com/workshop/neurips-2021#Workshop-Papers. Under review at a conference

openalex publication_date 2021/12/22 · openalex created_date 2022/05/05 · arxiv created 2022/06/27 · arxiv updated 2022/06/28 · openalex updated_date 2026/07/28

Abstract

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using any human data. Although such agents can be obtained through self-play training, they can suffer significantly from distributional shift when paired with unencountered partners, such as humans. To mitigate this distributional shift, we propose Maximum Entropy Population-based training (MEP). In MEP, agents in the population are trained with our derived Population Entropy bonus to promote both pairwise diversity between agents and individual diversity of agents themselves, and a common best agent is trained by paring with agents in this diversified population via prioritized sampling. The prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game environment, with partners being human proxy models and real humans. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.

Cited by

• Adaptively Coordinating with Novel Partners via Learned Latent Strategies
• Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games
• Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards
• Voting-Bloc Entropy: A New Metric for DAO Decentralization
• Virtual Agent Economies
• CausalPlan: Empowering Efficient LLM Multi-Agent Collaboration Through Causality-Driven Planning
• The Yokai Learning Environment: Tracking Beliefs Over Space and Time
• Unsupervised Partner Design Enables Robust Ad-hoc Teamwork
• Moving Out: Physically-grounded Human-AI Collaboration
• Modeling Latent Partner Strategies for Adaptive Zero-Shot Human-Agent Collaboration
• CooT: Learning to Coordinate In-Context with Coordination Transformers
• Efficient Generation of Diverse Cooperative Agents with World Models
• Towards Zero-Shot Coordination between Teams of Agents: The N-XPlay Framework
• MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning
• Implicitly Aligning Humans and Autonomous Agents through Shared Task Abstractions
• Enhancing Sample Efficiency in Multi-Agent RL with Uncertainty Quantification and Selective Exploration
• ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork
• Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors
• Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story
• Improving Human-AI Coordination through Online Adversarial Training and Generative Models
• Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

Related

• A Hierarchical Approach to Population Training for Human-AI Collaboration
• Learning to Cooperate with Humans using Generative Agents
• A Regularized Opponent Model with Maximum Entropy Objective
• Maximum Entropy Heterogeneous-Agent Reinforcement Learning
• Efficient Reinforcement Learning for Zero-Shot Coordination in Evolving Games