RRM: Robust Reward Model Training Mitigates Reward Hacking

2024/09/20 by Tianqi Liu, Liu, Tianqi, Wei Xiong +33 · 37 citations
Psychology · #Artificial intelligence #Cognitive psychology #Computation and Language (cs.CL) #Computer science #Computer security #FOS: Computer and information sciences #Geography #Hacker #Human-Automation Interaction and Safety #Psychology #Training (meteorology)

paper · pdf · doi:10.48550/arxiv.2409.13156

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

Cited by

• Mitigating Length Bias in RLHF through a Causal Lens
• AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
• Environment Scaling for Interactive Agentic Experience Collection: A Survey
• Reward Models are Metrics in a Trench Coat
• Rectifying Shortcut Behaviors in Preference-based Reward Learning
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
• APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
• Diversity-Incentivized Exploration for Versatile Reasoning
• RELISH: LLM REgression with a Latent Iterative State Head
• Virtual Agent Economies
• Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
• DEPTH: Hallucination-Free Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement
• Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction
• CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models in Mathematical Reasoning
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
• Intra-Trajectory Consistency for Reward Modeling
• Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
• Pre-Trained Policy Discriminators are General Reward Models
• Activation Reward Models for Few-Shot Model Alignment
• Inference-Time Reward Hacking in Large Language Models
• Preference Learning for AI Alignment: a Causal Perspective
• Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
• Learning a Pessimistic Reward Model in RLHF
• Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
• Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
• Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
• On the Robustness of Reward Models for Language Model Alignment
• Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
• HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
• Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
• Energy-Based Reward Models for Robust Language Model Alignment
• MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
• Adversarial Training of Reward Models

Related

• RewardBench: Evaluating Reward Models for Language Modeling
• WARM: On the Benefits of Weight Averaged Reward Models
• Activation Reward Models for Few-Shot Model Alignment
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
• Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling