A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning

2017/11/02 by Marc Lanctot, Vinícius Zambaldi, Vinicius Zambaldi +16 · 1 voice · 20 citations
Computer Science · #Artificial Intelligence in Games #Evolutionary Algorithms and Applications #Reinforcement Learning in Robotics #cs.AI #cs.GT #cs.LG #cs.MA

paper · pdf · doi:10.48550/arxiv.1711.00832

openalex publication_date 2017/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

To achieve general intelligence, agents must learn how to interact with others in a shared environment: this is the challenge of multiagent reinforcement learning (MARL). The simplest form is independent reinforcement learning (InRL), where each agent treats its experience as part of its (non-stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe an algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game-theoretic analysis to compute meta-strategies for policy selection. The algorithm generalizes previous ones such as InRL, iterated best response, double oracle, and fictitious play. Then, we present a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in two partially observable settings: gridworld coordination games and poker.

Citations

• Evolutionary stable strategies and game dynamics
• Action-Conditional Video Prediction using Deep Networks in Atari Games

Cited by

• CAP-DO: Learned Contextual Action Proposals for Certified Double-Oracle Solving Across Related Zero-Sum Games
• Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models
• Considering the Difference in Utility Functions of Team Players in Adversarial Team Games
• StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models
• R2PS: Worst-Case Robust Real-Time Pursuit Strategies under Partial Observability
• DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play
• Adaptively Coordinating with Novel Partners via Learned Latent Strategies
• Deep (Predictive) Discounted Counterfactual Regret Minimization
• Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces
• Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
• Nash Policy Gradient: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibria
• Look-ahead Reasoning with a Learned Model in Imperfect Information Games
• Downside Risk-Aware Equilibria for Strategic Decision-Making
• Beyond Game Theory Optimal: Profit-Maximizing Poker Agents for No-Limit Holdem
• Zero-sum turn games using Q-learning: finite computation with security guarantees
• Randomized Feasibility-Update Algorithms for Stochastic Variational Inequality Problems
• Private Markovian Equilibrium in Stackelberg Markov Games for Smart Grid Demand Response
• PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense
• The Docking Game: Loop Self-Play for Fast, Dynamic, and Accurate Prediction of Flexible Protein-Ligand Binding

Discussions

• In the past few years I have mostly worked on game-theoretic MARL. A good representative is policy-space response oracles (PSRO): arxiv.org/abs/1711.00832 3/N [bsky, 8 points, 1 comments]

Related

• Game-Theoretic Multiagent Reinforcement Learning
• A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
• RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learning
• Algorithms in Multi-Agent Systems: A Holistic Perspective from Reinforcement Learning and Game Theory