Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

2025/05/16 by Lang Feng, Jiahao Lin, Feng, Lang +9
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Evolutionary Game Theory and Cooperation #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2505.11100

openalex publication_date 2025/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.

Citations

• Learning Diverse Risk Preferences in Population-based Self-play
• Melting Pot 2.0
• Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination
• Collaborating with Humans without Human Data
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• RODE: Learning Roles to Decompose Multi-Agent Tasks
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret Learning in Games
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• "Other-Play" for Zero-Shot Coordination
• Dota 2 with Large Scale Deep Reinforcement Learning
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• Emergent Tool Use From Multi-Agent Autocurricula
• Compositionality decomposed: how do neural networks generalise?
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning
• Quantifying Generalization in Reinforcement Learning
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Generalization and Regularization in DQN
• Assessing Generalization in Deep Reinforcement Learning
• A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
• Learning with Opponent-Learning Awareness
• Proximal Policy Optimization Algorithms
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• The Option-Critic Architecture
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Distilling the Knowledge in a Neural Network

Related

• RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learning
• Policy Distillation and Value Matching in Multiagent Reinforcement Learning
• Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework
• A Survey on Self-play Methods in Reinforcement Learning
• A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning