Absolute Zero: Reinforced Self-play Reasoning with Zero Data

2025/05/06 by Andrew Zhao, Yiran Wu, Yilong Wu +20 · 23 voices · 76 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Multimodal Machine Learning Applications #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2505.03335

openalex publication_date 2025/05/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.

Citations

• VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks
• Boundless Socratic Learning with Language Games
• Prover-Verifier Games improve legibility of LLM outputs
• Self-Rewarding Language Models
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• Training language models to follow instructions with human feedback
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Distilling the Knowledge in a Neural Network

Cited by

• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
• Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning
• Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
• Dr. Zero: Self-Evolving Search Agents without Training Data
• Adaptive Time Series Reasoning via Segment Selection
• EvoSQL: Memory-Augmented Critic-Generator Co-Evolution for Text-to-SQL
• AI Must Embrace Specialization via Superhuman Adaptable Intelligence
• Bootstrapping Task Spaces for Self-Improvement
• R-Zero: Self-Evolving Reasoning LLM from Zero Data
• REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
• Outcome-based Reinforcement Learning to Predict the Future
• Measuring General Intelligence with Generated Games
• Reinforcement Learning via Self-Distillation
• CME-CAD: Heterogeneous Collaborative Multi-Expert Reinforcement Learning for CAD Code Generation
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• Safety Alignment of LMs via Non-cooperative Games
• TongSIM: A General Platform for Simulating Intelligent Machines
• Propose, Solve, Verify: Self-Play Through Formal Verification
• Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
• FrontierCS: Evolving Challenges for Evolving Intelligence
• MOA: Multi-Objective Alignment for Role-Playing Agents
• Probing the effectiveness of World Models for Spatial Reasoning through Test-time Scaling
• AI & Human Co-Improvement for Safer Co-Superintelligence
• Guided Self-Evolving LLMs with Minimal Human Supervision
• JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
• Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
• VisPlay: Self-Evolving Vision-Language Models from Images
• GPS: General Per-Sample Prompter
• P1: Mastering Physics Olympiads with Reinforcement Learning
• Better LLM Reasoning via Dual-Play
• Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis for Large Reasoning Models
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
• Adapting Web Agents with Synthetic Supervision
• Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials
• Towards Understanding Self-play for LLM Reasoning
• Autodata: An agentic data scientist to create high quality synthetic data
• Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• SPICE: Self-Play In Corpus Environments Improves Reasoning
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution
• Search Self-play: Pushing the Frontier of Agent Capability without Supervision
• EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learning
• Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning
• Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
• UniCode: A Framework for Generating High Quality Competitive Coding Problems
• Towards Agentic Self-Learning LLMs in Search Environment
• Spotlight on Token Perception for Multimodal Reinforcement Learning
• VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play
• AutoCode: LLMs as Problem Setters for Competitive Programming
• LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
• Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolution
• PIPer: On-Device Environment Setup via Online Reinforcement Learning
• Correct Reasoning Paths Visit Shared Decision Pivots
• Learning to Reason with Mixture of Tokens
• PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning
• SkillMentor: LLM Agent Self-Evolution via Learning Blind-Spot Diagnosis
• Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Language Self-Play For Data-Free Training
• Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents
• Learning to Generate Unit Test via Adversarial Reinforcement Learning
• Trust but Verify! A Survey on Verification Design for Test-time Scaling
• Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedback
• Sample-efficient LLM Optimization with Reset Replay
• Self-Questioning Language Models
• Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models
• CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks

Discussions

• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [hn, 88 points, 19 comments]
• seems interesting, perhaps even promising arxiv.org/pdf/2505.03335 [bsky, 20 points, 2 comments]
• hahahahah we're so fucked arxiv.org/pdf/2505.03335 > This underscores the need for a new paradigm that begins to explore possibilities beyond the constraints of human-designed tasks and prepares for a [bsky, 4 points, 1 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [hn, 3 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [hn, 3 points, 0 comments]
• I had Claude compare your post to the Absolute Zero paper from yesterday and its conclusion was “Blog Post as Prophecy”… arxiv.org/pdf/2505.03335 [bsky, 3 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [hn, 3 points, 2 comments]
• Real Vernor Vinge vibes from this paper that learns to code from scratch via RL "Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited [bsky, 2 points, 0 comments]
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data https://arxiv.org/abs/2505.03335v2 #AI #SelfPlayReasoning [bsky, 2 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [hn, 2 points, 0 comments]
• oi camarada eu venho aqui fazer um pedido atrevido pra vc (sigo o canal faz um tempo), tem chance de vc cobrir isso aqui? 🙏 arxiv.org/pdf/2505.03335 [bsky, 1 points, 0 comments]
• ⚡ Hackernews Top story: Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [bsky, 0 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data https://arxiv.org/abs/2505.03335 (https://news.ycombinator.com/item?id=43951985) [bsky, 0 points, 0 comments]
• Absolute Zero: the one where the model evolves through self-play without relying on external data. arxiv.org/abs/2505.03335 [bsky, 0 points, 1 comments]
• AI so smart it invents logic from nothing—too bad it’s arguing with itself about whether 2+2 is 5. www.arxiv.org/abs/25... [bsky, 0 points, 0 comments]
• AbsoluteZero: Reinforced Self-play Reasoningwith Zero Data arxiv.org/pdf/2505.03335 [bsky, 0 points, 0 comments]
• 今日読んだ論文 Absolute Zero: Reinforced Self-play Reasoning with Zero Data arxiv.org/abs/2505.03335 コーディング分野で、LLMが自分でタスク提案して自分で問いて強化学習をしていくという手法。コンセプト時点では抽象的に書かれているが、具体化されていくについて、やはりコーディング分野特有の性質（インタープリターで提 [bsky, 0 points, 0 comments]
• AbsoluteZero: Reinforced Self-play Reasoningwith Zero Data https://arxiv.org/pdf/2505.03335 [bsky, 0 points, 0 comments]
• Interesante paper: Este trabajo introduce un nuevo enfoque, llamado Absolute Zero, para entrenar modelos de razonamiento sin necesidad de datos etiquetados por humanos. arxiv.org/abs/2505.03335 [bsky, 0 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data https://arxiv.org/abs/2505.03335 (https://news.ycombinator.com/item?id=43951985) [bsky, 0 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data [bsky, 0 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data https://arxiv.org/abs/2505.03335 [bsky, 0 points, 0 comments]
• Absolute Zero: Reinforced Self-Play Reasoning with Zero Data https://arxiv.org/abs/2505.03335 [bsky, 0 points, 0 comments]

Related

• Zero Reinforcement Learning Towards General Domains
• R-Zero: Self-Evolving Reasoning LLM from Zero Data
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
• Reinforcing General Reasoning without Verifiers