2021/06/05 by Ming Zhou, Ziyu Wan, Zhou, Ming +15 · 25 citations
Computer Science · #Abstraction #Artificial Intelligence (cs.AI) #Artificial Intelligence in Games #Artificial intelligence #Computer science #Database #Distributed computing #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine learning #Multiagent Systems (cs.MA) #Parallel computing #Population #Reinforcement Learning in Robotics #Reinforcement learning #Scalability #Speedup #cs.AI #cs.MA
paper · pdf · doi:10.48550/arxiv.2106.07551
published in arXiv (Cornell University) (Cornell University) · 24 pages, 17 figures, 5 tables
arxiv created 2021/06/05 · openalex publication_date 2021/06/05 · arxiv updated 2021/06/15 · openalex created_date 2021/09/13 · openalex updated_date 2026/07/28
Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms, which produces a self-generated sequence of tasks arising from the coupled population dynamics. By leveraging auto-curricula to induce a population of distinct emergent strategies, PB-MARL has achieved impressive success in tackling multi-agent tasks. Despite remarkable prior arts of distributed RL frameworks, PB-MARL poses new challenges for parallelizing the training frameworks due to the additional complexity of multiple nested workloads between sampling, training and evaluation involved with heterogeneous policy interactions. To solve these problems, we present MALib, a scalable and efficient computing framework for PB-MARL. Our framework is comprised of three key components: (1) a centralized task dispatching model, which supports the self-generated tasks and scalable training with heterogeneous policy combinations; (2) a programming architecture named Actor-Evaluator-Learner, which achieves high parallelism for both training and sampling, and meets the evaluation requirement of auto-curriculum learning; (3) a higher-level abstraction of MARL training paradigms, which enables efficient code reuse and flexible deployments on different distributed computing paradigms. Experiments on a series of complex tasks such as multi-agent Atari Games show that MALib achieves throughput higher than 40K FPS on a single machine with 32 CPU cores; 5x speedup than RLlib and at least 3x speedup than OpenSpiel in multi-agent training tasks. MALib is publicly available at https://github.com/sjtu-marl/malib.