Why is Your Language Model a Poor Implicit Reward Model?

2025/07/10 by Noam Razin, Yong Lin, Razin, Noam +5
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.07981

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reward models are key to language model post-training and inference pipelines. Conveniently, recent work showed that every language model defines an implicit reward model (IM-RM), without requiring any architectural changes. However, such IM-RMs tend to generalize worse, especially out-of-distribution, compared to explicit reward models (EX-RMs) that apply a dedicated linear head over the hidden representations of a language model. The existence of a generalization gap is puzzling, as EX-RMs and IM-RMs are nearly identical. They can be trained using the same data, loss function, and language model, and differ only in how the reward is computed. Toward a fundamental understanding of the implicit biases underlying different reward model types, we investigate the root cause of this gap. Our main finding, backed by theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. Consequently, they often generalize worse than EX-RMs under token-level distribution shifts, as well as in-distribution. Furthermore, we provide evidence against alternative hypotheses for the generalization gap. Most notably, we challenge the claim that IM-RMs struggle in tasks where generation is harder than verification because they can operate both as a verifier and a generator. Overall, our results highlight that seemingly minor design choices can substantially impact the generalization behavior of reward models.

Citations

• RewardBench 2: Advancing Reward Model Evaluation
• The Rich and the Simple: On the Implicit Bias of Adam and SGD
• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
• On the Robustness of Reward Models for Language Model Alignment
• What Makes a Reward Model a Good Teacher? An Optimization Perspective
• reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
• Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
• InfAlign: Inference-aware language model alignment
• Qwen2.5 Technical Report
• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
• Optimal Design for Reward Modeling in RLHF
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• On The Global Convergence Of Online RLHF With Neural Parametrization
• How to Evaluate Reward Models for RLHF
• The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels
• RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
• The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models
• Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?
• Evaluating Robustness of Reward Models for Mathematical Reasoning
• On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Can DPO Learn Diverse Human Values? A Theoretical Scaling Law
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Learning Dynamics of LLM Finetuning
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Nemotron-4 340B Technical Report
• The Implicit Bias of Adam on Separable Data
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• RLHF Workflow: From Reward Modeling to Online RLHF
• D2PO: Discriminator-Guided DPO with Response Evaluation Models
• RewardBench: Evaluating Reward Models for Language Modeling
• A Survey on Data Selection for Language Models
• Active Preference Optimization for Sample Efficient RLHF
• Reward Generalization in RLHF: A Topological Perspective
• Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
• Reinforcement Learning from Human Feedback with Active Queries
• Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
• LESS: Selecting Influential Data for Targeted Instruction Tuning
• Transforming and Combining Rewards for Aligning Large Language Models
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
• The Linear Representation Hypothesis and the Geometry of Large Language Models
• Vanishing Gradients in Reinforcement Finetuning of Language Models
• Making RL with Preference-based Feedback Efficient via Randomization
• Representation Engineering: A Top-Down Approach to AI Transparency
• Reinforced Self-Training (ReST) for Language Modeling
• Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems
• Is RLHF More Difficult than Standard RL?
• Let's Verify Step by Step
• Provable Reward-Agnostic Preference-Based Reinforcement Learning
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Provable Offline Preference-Based Reinforcement Learning
• RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• GPT-4 Technical Report
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
• Discovering Language Model Behaviors with Model-Written Evaluations
• Solving math word problems with process- and outcome-based feedback
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets
• On the Implicit Bias in Deep-Learning Algorithms
• Training language models to follow instructions with human feedback
• Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks
• Training Verifiers to Solve Math Word Problems
• Implicit Regularization in Tensor Factorization
• Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Estimating Training Data Influence by Tracing Gradient Descent
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Fine-Tuning Language Models from Human Preferences
• Implicit Regularization in Deep Matrix Factorization
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural\n Networks
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• The Implicit Bias of Gradient Descent on Separable Data
• Attention Is All You Need
• Implicit Regularization in Matrix Factorization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
• UltraFeedback: Boosting Language Models with Scaled AI Feedback

Related

• Reward Models are Metrics in a Trench Coat
• RewardBench: Evaluating Reward Models for Language Modeling
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• RewardAnything: Generalizable Principle-Following Reward Models
• Generalist Reward Models: Found Inside Large Language Models