RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

2025/11/10 by Zhiyuan Zeng, Zeng, Zhiyuan, Hamish Ivison +31 · 5 citations
Computer Science · #Topic Modeling #Reinforcement Learning in Robotics #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.07317

Abstract

We introduce Reinforcement Learning (RL) with Adaptive Verifiable Environments (RLVE), an approach using verifiable environments that procedurally generate problems and provide algorithmically verifiable rewards, to scale up RL for language models (LMs). RLVE enables each verifiable environment to dynamically adapt its problem difficulty distribution to the policy model's capabilities as training progresses. In contrast, static data distributions often lead to vanishing learning signals when problems are either too easy or too hard for the policy. To implement RLVE, we create RLVE-Gym, a large-scale suite of 400 verifiable environments carefully developed through manual environment engineering. Using RLVE-Gym, we show that environment scaling, i.e., expanding the collection of training environments, consistently improves generalizable reasoning capabilities. RLVE with joint training across all 400 environments in RLVE-Gym yields a 3.37% absolute average improvement across six reasoning benchmarks, starting from one of the strongest 1.5B reasoning LMs. By comparison, continuing this LM's original RL training yields only a 0.49% average absolute gain despite using over 3x more compute. We release our code publicly.

Citations

• SPICE: Self-Play In Corpus Environments Improves Reasoning
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Prompt Curriculum Learning for Efficient LLM Post-Training
• APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
• InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling
• Kimi K2: Open Agentic Intelligence
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization
• OpenThoughts: Data Recipes for Reasoning Models
• REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards
• SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond
• Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
• Self-Evolving Curriculum for LLM Reasoning
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models
• Qwen3 Technical Report
• Absolute Zero: Reinforced Self-play Reasoning with Zero Data
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• What Makes a Reward Model a Good Teacher? An Optimization Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• BIG-Bench Extra Hard
• Process Reinforcement through Implicit Rewards
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Qwen2.5 Technical Report
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• On Memorization of Large Language Models in Logical Reasoning
• DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
• The Need for a Big World Simulator: A Scientific Challenge for Continual Learning
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• SGLang: Efficient Execution of Structured Language Model Programs
• Vanishing Gradients in Reinforcement Finetuning of Language Models
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• One Embedder, Any Task: Instruction-Finetuned Text Embeddings
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Training language models to follow instructions with human feedback
• Prioritized Level Replay
• Automatic Curriculum Learning For Deep RL: A Short Survey
• Emergent Tool Use From Multi-Agent Autocurricula
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Cited by

• From Word to World: Can Large Language Models be Implicit Text-based World Models?
• ThetaEvolve: Test-time Learning on Open Problems
• Cognitive Foundations for Reasoning and Their Manifestation in LLMs
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

Related

• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
• GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning
• Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration