Generative Verifiers: Reward Modeling as Next-Token Prediction

2024/08/27 by Lunjun Zhang, Arian Hosseini, Zhang, Lunjun +9 · 103 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2408.15240

openalex publication_date 2024/08/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional test-time compute via majority voting for better verification. We demonstrate that GenRM outperforms discriminative, DPO verifiers, and LLM-as-a-Judge, resulting in large performance gains with Best-of-N, namely 5% → 45.3% on algorithmic tasks and 73% → 93.4% on GSM8K. In easy-to-hard generalization settings, we observe improvements of 28% → 44.6% on MATH, and 37.9% → 53.5% on MMLU abstract algebra. Furthermore, we find that training GenRM with synthetic verification rationales is sufficient to pick out subtle errors on math problems. Finally, we demonstrate that GenRM scales favorably with model size and test-time compute.

Cited by

• Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks
• EVE: A Generator-Verifier System for Generative Policies
• Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
• Population-Evolve: a Parallel Sampling and Evolutionary Method for LLM Math Reasoning
• Training LLMs with LogicReward for Faithful and Rigorous Reasoning
• AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
• Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
• Stepwise Think-Critique: A Unified Framework for Robust and Interpretable LLM Reasoning
• Estimating problem difficulty without ground truth using Large Language Model comparisons
• Differentiable Evolutionary Reinforcement Learning
• GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
• OPV: Outcome-based Process Verifier for Efficient Long Chain-of-Thought Verification
• IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation
• When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
• Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
• When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers
• Zero-Overhead Introspection for Adaptive Test-Time Compute
• Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search
• Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selection
• On Evaluating LLM Alignment by Evaluating LLMs as Judges
• EduMod-LLM: A Modular Approach for Designing Flexible and Transparent Educational Assistants
• From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
• Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
• PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
• AIA Forecaster: Technical Report
• RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks
• MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL
• Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
• LLM-as-a-Verifier: A General-Purpose Verification Framework
• PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
• Think Twice: Branch-and-Rethink Reasoning Reward Model
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
• The Virtues of Brevity: Avoid Overthinking in Parallel Test-Time Reasoning
• Language Ranker: A Lightweight Ranking framework for LLM Decoding
• Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
• VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data
• Dual-Weighted Reinforcement Learning for Generative Preference Modeling
• An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs
• Learning an Image Editing Model without Image Editing Pairs
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models
• Budget-aware Test-time Scaling via Discriminative Verification
• Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math
• Confidence as a Reward: Transforming LLMs into Reward Models
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• The Art of Scaling Reinforcement Learning Compute for LLMs
• A Survey on Parallel Reasoning
• KnowRL: Teaching Language Models to Know What They Know
• Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimization
• FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
• Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
• Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
• Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
• A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models
• Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
• Lateral Tree-of-Thoughts Surpasses ToT by Incorporating Logically-Consistent, Low-Utility Candidates
• mR3: Multilingual Rubric-Agnostic Reward Reasoning Models
• Rethinking Reward Models for Multi-Domain Test-Time Scaling
• Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning
• Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
• From Faithfulness to Correctness: Generative Reward Models that Think Critically
• Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
• Who's Your Judge? On the Detectability of LLM-Generated Judgments
• Reference-Free Rating of LLM Responses via Latent Information
• ContextPRM: Leveraging Contextual Coherence for multi-domain Test-Time Scaling
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
• References Improve LLM Alignment in Non-Verifiable Domains
• Variation in Verification: Understanding Verification Dynamics in Large Language Models
• SCAN: Self-Denoising Monte Carlo Annotation for Robust Process Reward Learning
• Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing
• Reverse-Engineered Reasoning for Open-Ended Generation
• GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning
• Reinforcement Learning for Machine Learning Engineering Agents
• LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
• StepWiser: Stepwise Generative Judges for Wiser Reasoning
• Trust but Verify! A Survey on Verification Design for Test-time Scaling
• Improving Value-based Process Verifier via Low-Cost Variance Reduction
• ReviewRL: Towards Automated Scientific Review with RL
• WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Library
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs
• CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
• Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillation
• Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
• Uncertainty-Based Methods for Automated Process Reward Data Construction and Output Aggregation in Mathematical Reasoning
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• G-Core: A Simple, Scalable and Balanced RLHF Trainer
• Libra: Assessing and Improving Reward Model by Learning to Think
• UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities
• V₁: Unifying Generation and Self-Verification for Parallel Reasoners

Related

• When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoning
• Solve-Detect-Verify: Inference-Time Scaling with Flexible Generative Verifier
• Shrinking the Generation-Verification Gap with Weak Verifiers
• Scaling Generative Verifiers For Natural Language Mathematical Proof Verification And Selection
• GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning