K-Level Policy Gradients for Multi-Agent Reinforcement Learning

2025/09/15 by Aryaman Reddi, Gabriele Tiboni, Reddi, Aryaman +5 · 1 citation
Computer Science · #Convergence (economics) #Implementation #Iterated function #Monotonic function #Policy learning #Reinforcement Learning in Robotics #Reinforcement learning

paper · pdf · doi:10.48550/arxiv.2509.12117

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/09/15 · openalex created_date 2025/10/12 · openalex updated_date 2026/08/05

Abstract

Actor-critic algorithms for deep multi-agent reinforcement learning (MARL) typically employ a policy update that responds to the current strategies of other agents. While being straightforward, this approach does not account for the updates of other agents at the same update step, resulting in miscoordination. In this paper, we introduce the K-Level Policy Gradient (KPG), a method that recursively updates each agent against the updated policies of other agents, speeding up the discovery of effective coordinated policies. We theoretically prove that KPG with finite iterates achieves monotonic convergence to a local Nash equilibrium under certain conditions. We provide principled implementations of KPG by applying it to the deep MARL algorithms MAPPO, MADDPG, and FACMAC. Empirically, we demonstrate superior performance over existing deep MARL algorithms in StarCraft II and multi-agent MuJoCo.

Citations

• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
• Applications of Multi-Agent Deep Reinforcement Learning Communication in Network Management: A Survey
• Scaling Opponent Shaping to High Dimensional Games
• Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
• Game Theoretic Decision Making by Actively Learning Human Intentions Applied on Autonomous Driving
• Recursive Reasoning in Minimax Games: A Level k Gradient Play Method
• Proximal Learning With Opponent-Learning Awareness
• COLA: Consistent Learning with Opponent-Learning Awareness
• Generative Adversarial Networks
• The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• Parameter-Based Value Functions
• Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey
• Driver Modeling through Deep Reinforcement Learning and Behavioral Game Theory
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• The StarCraft Multi-Agent Challenge
• The Hanabi Challenge: A New Frontier for AI Research
• Learning with Opponent-Learning Awareness
• Proximal Policy Optimization Algorithms
• Value-Decomposition Networks For Cooperative Multi-Agent Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Counterfactual Multi-Agent Policy Gradients
• Opponent Modeling in Deep Reinforcement Learning
• Adam: A Method for Stochastic Optimization
• JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

Cited by

• The Auton Agentic AI Framework

Related

• Settling the Variance of Multi-Agent Policy Gradients
• FACMAC: Factored Multi-Agent Centralised Policy Gradients
• 3DPG: Distributed Deep Deterministic Policy Gradient Algorithms for Networked Multi-Agent Systems
• Off-Policy Multi-Agent Decomposed Policy Gradients
• Scalable Centralized Deep Multi-Agent Reinforcement Learning via Policy Gradients