Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

2025/03/31 by Jingcheng Hu, Y. Zhang, Yinmin Zhang +11 · 1 voice · 293 citations
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Reinforcement Learning in Robotics #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2503.24290

openalex publication_date 2025/03/31 · arxiv published 2025/03/31 · arxiv updated 2025/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce Open-Reasoner-Zero, the first open source implementation of large-scale reasoning-oriented RL training on the base model focusing on scalability, simplicity and accessibility. Through extensive experiments, we demonstrate that a minimalist approach, vanilla PPO with GAE (λ=1, γ=1) and straightforward rule-based rewards, without any KL regularization, is sufficient to scale up both benchmark performance and response length, replicating the scaling phenomenon observed in DeepSeek-R1-Zero. Using the same base model, Qwen2.5-32B base, as DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance across AIME2024, MATH500, and GPQA Diamond, while demonstrating remarkable efficiency, requiring only 1/10 of the training steps compared to the DeepSeek-R1-Zero pipeline. Moreover, our analysis not only covers training dynamics and ablation for critical design choices, but also quantitatively shows how the learned critic in Reasoner-Zero training effectively identifies and devalues repetitive response patterns, yielding more robust advantage estimations and enhancing training stability. Embracing the principles of open-source, we release our source code, training data, and various model weights, fostering reproducibility and encouraging further exploration of the properties of related models.

Cited by

• Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations
• On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
• LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
• Multi-Task GRPO: Reliable LLM Reasoning Across Tasks
• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
• Trust-Region Adaptive Policy Optimization
• INTELLECT-3: Technical Report
• Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
• RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training
• Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learning
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learning
• SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
• Rectifying LLM Thought from Lens of Optimization
• ESPO: Entropy Importance Sampling Policy Optimization
• Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
• JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
• ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learning
• SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
• EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control
• SERL: Self-Examining Reinforcement Learning on Open-Domain
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View
• What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
• FLEX: Continuous Agent Evolution via Forward Learning from Experience
• QueStER: Query Specification for Generative keyword-based Retrieval
• Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
• Think Outside the Policy: In-Context Steered Policy Optimization
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning
• Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• Scheduling Your LLM Reinforcement Learning with Reasoning Trees
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
• PACR: Progressively Ascending Confidence Reward for LLM Reasoning
• Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
• Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning
• Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
• A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
• Extracting alignment data in open models
• Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs
• Deep Self-Evolving Reasoning
• AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction
• Soundness-Aware Level: A Microscopic Signature that Predicts LLM Reasoning Potential
• The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• Reasoning with Sampling: Your Base Model is Smarter Than You Think
• SimKO: Simple Pass@K Policy Optimization
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Reinforced Preference Optimization for Recommendation
• Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregation
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
• Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
• RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
• On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
• Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• DARO: Difficulty-Aware Reweighting Policy Optimization
• GCPO: When Contrast Fails, Go Gold
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation
• Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• When Thinking Drifts: Evidential Grounding for Robust Video Reasoning
• Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
• Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
• Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewards
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
• GRPO-λ: Credit Assignment improves LLM Reasoning
• Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
• Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling
• Diversity-Incentivized Exploration for Versatile Reasoning
• IRIS: Intrinsic Reward Image Synthesis
• Rethinking Entropy Regularization in Large Reasoning Models
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
• Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
• Taming Masked Diffusion Language Models via Consistency Trajectory Reinforcement Learning with Fewer Decoding Step
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking
• Structured In-context Environment Scaling for Large Language Model Reasoning
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Variational Reasoning for Language Models
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• RLP: Reinforcement as a Pretraining Objective
• A State-of-the-Art SQL Reasoning Model using RLVR
• ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning
• Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
• ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
• Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning
• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute
• MAPO: Mixed Advantage Policy Optimization
• PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• GPO: Learning from Critical Steps to Improve LLM Reasoning
• TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference
• Omne-R1: Learning to Reason with Memory for Multi-hop Question Answering
• VL Norm: Rethink Loss Aggregation in RLVR
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
• Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors
• TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning
• Towards a Unified View of Large Language Model Post-Training
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
• OwkinZero: Accelerating Biological Discovery with AI
• Kwai Keye-VL 1.5 Technical Report
• DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization
• Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
• Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation
• MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
• CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
• Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
• Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning
• EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• UR²: Unify RAG and Reasoning through Reinforcement Learning
• Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting
• Self-Questioning Language Models
• GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
• Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMs
• EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
• Geometric-Mean Policy Optimization
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
• StepFun-Prover Preview: Let's Think and Verify Step by Step
• Agentic Reinforced Policy Optimization
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization
• Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
• A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoning
• VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarks
• RePO: Replay-Enhanced Policy Optimization
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
• SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
• RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
• REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once
• VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
• Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
• From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
• Skywork-R1V3 Technical Report
• BlueLM-2.5-3B Technical Report
• FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
• Replacing thinking with tool usage enables reasoning in small language models
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
• StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason
• WebSailor: Navigating Super-human Reasoning for Web Agent
• RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
• Kwai Keye-VL Technical Report
• Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
• GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
• ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context
• Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
• Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning
• A Survey of Continual Reinforcement Learning
• Through the Valley: Path to Effective Long CoT Training for Small Language Models
• Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
• Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
• Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
• OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling
• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
• Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning
• RLPR: Extrapolating RLVR to General Domains without Verifiers
• Truncated Proximal Policy Optimization
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Tasks
• Socratic RL: A Novel Framework for Efficient Knowledge Acquisition through Iterative Reflection and Viewpoint Distillation
• How Far Are We from Optimal Reasoning Efficiency?
• Magistral
• Spurious Rewards: Rethinking Training Signals in RLVR
• Time Series Forecasting as Reasoning: A Slow-Thinking Approach with Reinforced LLMs
• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
• TreeRPO: Tree Relative Policy Optimization
• EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning
• BPO: Revisiting Preference Modeling in Direct Preference Optimization
• Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
• StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs
• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
• KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
• TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence
• QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
• Table-R1: Inference-Time Scaling for Table Reasoning
• X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains
• Diversity-Aware Policy Optimization for Large Language Model Reasoning
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
• DeepTheorem: Advancing LLM Reasoning for Theorem Proving Through Natural Language and Reinforcement Learning
• Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding
• From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
• The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• Reinforcing General Reasoning without Verifiers
• Thinker: Learning to Think Fast and Slow
• Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning
• Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
• What Can RL Bring to VLA Generalization? An Empirical Study
• Learning to Reason without External Rewards
• Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection
• SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond
• One-shot Entropy Minimization
• Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
• ARM: Adaptive Reasoning Model
• Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
• Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• Steering LLM Reasoning Through Bias-Only Adaptation
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
• Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models
• Generative RLHF-V: Learning Principles from Multi-modal Human Preference
• Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals
• Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration
• Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models
• KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning
• NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning
• TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning
• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
• Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
• Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning
• AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Momentum
• General-Reasoner: Advancing LLM Reasoning Across All Domains
• SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learning
• Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
• The Hallucination Tax of Reinforcement Finetuning
• Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings
• Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs
• DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
• Optimizing Anytime Reasoning via Budget Relative Policy Optimization
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
• DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
• CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models
• SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
• TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
• Is PRM Necessary? Problem-Solving RL Implicitly Induces PRM Capability in LLMs
• Attention-Steered Vision-Language Models for Sign Language Translation
• Uncertainty-Aware Simulation-Based Inference for Operations Research with Large Language Models
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
• MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
• DeepCritic: Deliberate Critique with Large Language Models
• Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning
• BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?
• The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
• 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models
• WebThinker: Empowering Large Reasoning Models with Deep Research Capability
• Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
• Weak-to-Strong Generalization via Direct On-Policy Distillation
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
• Fast-Slow Thinking GRPO for Large Vision-Language Model Reasoning
• PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning
• CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
• Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO
• Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
• TTRL: Test-Time Reinforcement Learning
• Tina: Tiny Reasoning Models via LoRA
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
• Learning to Reason under Off-Policy Guidance
• Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
• From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs
• Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
• DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
• GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
• Perception-R1: Pioneering Perception Policy with Reinforcement Learning
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
• A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
• Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization

Discussions

• 4. Open-Reasoner-Zero Open-source implementation of a minimalist RL training approach for LLMs, mirroring DeepSeek-R1-Zero’s effectiveness with fewer steps arxiv.org/abs/2503.24290 GitHub: github.com/ [bsky, 0 points, 1 comments]

Related

• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning
• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
• Apriel-1.5-OpenReasoner: RL Post-Training for General-Purpose and Efficient Reasoning