2020/06/08 by Thomas Anthony, Anthony, Thomas, Tom Eccles +25 · 2 voices · 2 citations
Computer Science · Economics, Econometrics and Finance · #Artificial Intelligence in Games #Reinforcement Learning in Robotics #Sports Analytics and Performance #cs.AI #cs.GT #cs.LG #cs.MA #stat.ML
paper · pdf · doi:10.48550/arxiv.2006.04635
openalex publication_date 2020/06/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.