Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers

2025/05/07 by Kusha Sareen, Sareen, Kusha, Morgane M Moss +7 · 11 citations
Computer Science · Decision Sciences · #Base (topology) #Explainable Artificial Intelligence (XAI) #Function (biology) #Generalization #Reinforcement Learning in Robotics #Reinforcement learning #Scaling #Semantic reasoner #Stock Market Forecasting Methods #Value (mathematics)

paper · pdf · doi:10.48550/arxiv.2505.04842

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. Yet if parallel test-time compute is already part of the deployment plan, training should be designed to support it. In this work, we propose RL^V that augments any ``value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL^V boosts MATH accuracy by over 20% with parallel sampling and enables 8-32× efficient test-time compute scaling compared to the base RL method. RL^V also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL^V achieves 1.2-1.6× higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model. More broadly, RL^V instantiates the principle of co-training for test-time scaling: jointly optimizing for task performance and a capability useful at inference, using data that RL training already produces.

Citations

• Scaling Test-Time Compute Without Verification or RL is Suboptimal
• Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
• SETS: Leveraging Self-Verification and Self-Correction for Improved Test-Time Scaling
• s1: Simple test-time scaling
• The Lessons of Developing Process Reward Models in Mathematical Reasoning
• Free Process Rewards without Process Labels
• Generative Reward Models
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Critique-out-Loud Reward Models
• AI-Assisted Generation of Difficult Math Questions
• Improve Mathematical Reasoning in Language Models by Automated Process Supervision
• Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
• V-STaR: Training Verifiers for Self-Taught Reasoners
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning
• Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
• Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Let's Verify Step by Step
• Constitutional AI: Harmlessness from AI Feedback
• Solving math word problems with process- and outcome-based feedback
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• Learning to summarize from human feedback
• Proximal Policy Optimization Algorithms
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation

Cited by

• When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• The Alignment Waltz: Jointly Training Agents to Collaborate for Safety
• On the Role of Temperature Sampling in Test-Time Scaling
• Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solving
• Trust but Verify! A Survey on Verification Design for Test-time Scaling
• Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
• V₁: Unifying Generation and Self-Verification for Parallel Reasoners
• Kevin: Multi-Turn RL for Generating CUDA Kernels
• Incentivizing LLMs to Self-Verify Their Answers

Related

• Scaling Test-Time Compute Without Verification or RL is Suboptimal
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
• ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding
• RLPR: Extrapolating RLVR to General Domains without Verifiers
• Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning