Mastering the Game of Go with Self-play Experience Replay

2026/01/06 by Jingbin Liu, Xuechun Wang · 1 voice
Computer Science · #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.2601.03306

Abstract

The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.

Citations

• Mastering the game of Stratego with model-free multiagent reinforcement learning
• Training language models to follow instructions with human feedback
• Soft Q Network
• Dota 2 with Large Scale Deep Reinforcement Learning
• ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero
• Soft Actor-Critic Algorithms and Applications
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Equivalence Between Policy Gradients and Soft Q-Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Mastering the game of Go with deep neural networks and tree search
• Deep Residual Learning for Image Recognition
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Playing Atari with Deep Reinforcement Learning

Discussions

• Someone got Q learning to be stable for 5 months, that’s crazy arxiv.org/abs/2601.033... [bsky, 1 points, 0 comments]

Related

• Assessing the Potential of Classical Q-learning in General Game Playing
• Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations
• Combining Off and On-Policy Training in Model-Based Reinforcement Learning
• Improve Value Estimation of Q Function and Reshape Reward with Monte Carlo Tree Search
• Targeted Search Control in AlphaZero for Effective Policy Improvement