R-Zero: Self-Evolving Reasoning LLM from Zero Data

2025/08/07 by Chengsong Huang, Wenhao Yu, Huang, Chengsong +15 · 12 voices · 50 citations
#cs.LG #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2508.05004

Abstract

Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.

Citations

• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• AI models collapse when trained on recursively generated data
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Cited by

• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
• Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning
• Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
• Dr. Zero: Self-Evolving Search Agents without Training Data
• Adaptive Time Series Reasoning via Segment Selection
• EvoSQL: Memory-Augmented Critic-Generator Co-Evolution for Text-to-SQL
• LaRec: Unleashing LLM-based Latent Reasoning for Generative Recommendation
• Reinforcement Learning via Self-Distillation
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering
• Propose, Solve, Verify: Self-Play Through Formal Verification
• UniRel: Relation-Centric Knowledge Graph Question Answering with RL-Tuned LLM Reasoning
• PretrainZero: Reinforcement Active Pretraining
• Guided Self-Evolving LLMs with Minimal Human Supervision
• Syn-GRPO: Self-Evolving Data Synthesis for MLLM Perception Reasoning
• UNSEEN: Enhancing Dataset Pruning from a Generalization Perspective
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
• VisPlay: Self-Evolving Vision-Language Models from Images
• Better LLM Reasoning via Dual-Play
• Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis for Large Reasoning Models
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Towards Understanding Self-play for LLM Reasoning
• Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Automating Benchmark Design
• SPICE: Self-Play In Corpus Environments Improves Reasoning
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution
• Search Self-play: Pushing the Frontier of Agent Capability without Supervision
• MENTOR: A Reinforcement Learning Framework for Enabling Tool Use in Small Models via Teacher-Optimized Rewards
• Deep Self-Evolving Reasoning
• Towards Agentic Self-Learning LLMs in Search Environment
• LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
• GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare
• Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs
• Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense
• RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
• CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
• NanoFlux: Adversarial Dual-LLM Evaluation and Distillation For Multi-Domain Reasoning
• ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning
• Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation
• Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
• Discovering New Theorems via LLMs with In-Context Proof Learning in Lean
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Language Self-Play For Data-Free Training
• Parallel-R1: Towards Parallel Thinking via Reinforcement Learning
• Self-Rewarding Vision-Language Model via Reasoning Decomposition
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• One Token to Fool LLM-as-a-Judge

Discussions

• R-Zero: Self-Evolving Reasoning LLM from Zero Data [hn, 121 points, 61 comments]
• Yeesh. AI: We don't need no stinking humans. We can train ourselves. "Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, a [bsky, 17 points, 7 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data [hn, 3 points, 1 comments]
• R-Zero: Codes for R-Zero: Self-Evolving Reasoning LLM from Zero Data [hn, 2 points, 1 comments]
• R-Zero (arxiv.org/abs/2508.05004) for LLMs specifically PAIRED (arxiv.org/abs/2012.02096) and ReMiDi (arxiv.org/abs/2402.12284) for RL more generally [bsky, 1 points, 1 comments]
• A Challenger and a Solver. arxiv.org/abs/2508.05004 [bsky, 1 points, 2 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data https://arxiv.org/abs/2508.05004 (https://news.ycombinator.com/item?id=45192194) [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: R-Zero: Self-Evolving Reasoning LLM from Zero Data [bsky, 0 points, 0 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data https://arxiv.org/abs/2508.05004 (https://news.ycombinator.com/item?id=45192194) [bsky, 0 points, 0 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data https://arxiv.org/abs/2508.05004 https://news.ycombinator.com/item?id=45192194 [bsky, 0 points, 0 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data [bsky, 0 points, 0 comments]
• R-Zero: Self-Evolving Reasoning LLM from Zero Data https://arxiv.org/abs/2508.05004 [bsky, 0 points, 0 comments]

Related

• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
• L0: Reinforcement Learning to Become General Agents
• Zero Reinforcement Learning Towards General Domains
• Guided Self-Evolving LLMs with Minimal Human Supervision