SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

2025/09/03 by Mukherjee, Arpan, Bullo, Marcello, Gündüz, Deniz
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2509.03672

Abstract

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed \em SharedRep-RLHF. At its core, SharedRep-RLHF learns and leverages \em shared traits in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep-RLHF. Experiments across diverse natural language tasks showcase the effectiveness of SharedRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.

Citations

• Robust Multi-Objective Controlled Decoding of Large Language Models
• Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration
• When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
• Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
• PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
• Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
• Group Robust Preference Optimization in Reward-free RLHF
• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
• Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
• Iterative Reasoning Preference Optimization
• Active Preference Optimization for Sample Efficient RLHF
• MaxMin-RLHF: Alignment with Diverse Human Preferences
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
• Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning
• Whose Opinions Do Language Models Reflect?
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• A General Language Assistant as a Laboratory for Alignment
• Dueling RL: Reinforcement Learning with Trajectory Preferences
• Training Verifiers to Solve Math Word Problems
• Impact of Representation Learning in Linear Bandits
• Learning to summarize from human feedback
• Bandit Algorithms
• Proximal Policy Optimization Algorithms
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
• Aligning Large Language Models with Human: A Survey

Related

• MaxMin-RLHF: Alignment with Diverse Human Preferences
• Secrets of RLHF in Large Language Models Part II: Reward Modeling
• MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
• A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
• Accelerating RLHF Training with Reward Variance Increase