Quantile Regression for Distributional Reward Models in RLHF

2024/09/16 by Nicolai Dorka, Dorka, Nicolai · 26 citations
Mathematics · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2409.10164

openalex publication_date 2024/09/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning from human feedback (RLHF) has become a key method for aligning large language models (LLMs) with human preferences through the use of reward models. However, traditional reward models typically generate point estimates, which oversimplify the diversity and complexity of human values and preferences. In this paper, we introduce Quantile Reward Models (QRMs), a novel approach to reward modeling that learns a distribution over rewards instead of a single scalar value. Our method uses quantile regression to estimate a full, potentially multimodal distribution over preferences, providing a more powerful and nuanced representation of preferences. This distributional approach can better capture the diversity of human values, addresses label noise, and accommodates conflicting preferences by modeling them as distinct modes in the distribution. Our experimental results show that QRM outperforms comparable traditional point-estimate models on RewardBench. Furthermore, we demonstrate that the additional information provided by the distributional estimates can be utilized in downstream applications, such as risk-aware reinforcement learning, resulting in LLM policies that generate fewer extremely negative responses. Our code and model are released at https://github.com/Nicolinho/QRM.

Cited by

• RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
• When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models
• DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
• PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
• Ask a Strong LLM Judge when Your Reward Model is Uncertain
• Safety Game: Balancing Safe and Informative Conversations with Blackbox Agentic AI using LP Solvers
• Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
• Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs
• CTTS: Collective Test-Time Scaling
• Shrinking the Generation-Verification Gap with Weak Verifiers
• Reward Model Interpretability via Optimal and Pessimal Tokens
• PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization
• RewardBench 2: Advancing Reward Model Evaluation
• Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences
• Multi-Domain Explainability of Preferences
• Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
• Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?
• Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
• A Systematic Analysis of Base Model Choice for Reward Modeling
• Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
• Sandcastles in the Storm: Revisiting the (Im)possibility of Strong Watermarking
• Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
• Reward Models Inherit Value Biases from Pretraining
• Energy-Based Reward Models for Robust Language Model Alignment

Related

• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
• Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
• A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
• Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback