2020/12/22 by Pascal Leroy, Leroy, Pascal, Damien Ernst +9
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Reinforcement Learning in Robotics #cs.AI #cs.LG #cs.MA
paper · pdf · doi:10.48550/arxiv.2012.12062
To be published in AAAI-21 Workshop on Reinforcement Learning in Games
arxiv created 2020/12/22 · openalex publication_date 2020/12/22 · arxiv updated 2020/12/23 · openalex created_date 2021/01/05 · openalex updated_date 2026/07/28
This paper introduces four new algorithms that can be used for tackling multi-agent reinforcement learning (MARL) problems occurring in cooperative settings. All algorithms are based on the Deep Quality-Value (DQV) family of algorithms, a set of techniques that have proven to be successful when dealing with single-agent reinforcement learning problems (SARL). The key idea of DQV algorithms is to jointly learn an approximation of the state-value function V, alongside an approximation of the state-action value function Q. We follow this principle and generalise these algorithms by introducing two fully decentralised MARL algorithms (IQV and IQV-Max) and two algorithms that are based on the centralised training with decentralised execution training paradigm (QVMix and QVMix-Max). We compare our algorithms with state-of-the-art MARL techniques on the popular StarCraft Multi-Agent Challenge (SMAC) environment. We show competitive results when QVMix and QVMix-Max are compared to well-known MARL techniques such as QMIX and MAVEN and show that QVMix can even outperform them on some of the tested environments, being the algorithm which performs best overall. We hypothesise that this is due to the fact that QVMix suffers less from the overestimation bias of the Q function.