Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

2024/02/11 by Chenlu Ye, Ye, Chenlu, Wei Xiong +7 · 11 citations
Computer Science · Decision Sciences · Engineering · #Advanced Decision-Making Techniques #Evaluation and Optimization Models #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multi-Criteria Decision Making

paper · pdf · doi:10.48550/arxiv.2402.07314

openalex publication_date 2024/02/11 · openalex created_date 2024/02/14 · openalex updated_date 2026/08/01

Abstract

We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

Cited by

• Optimizing NetGPT via Routing-Based Synergy and Reinforcement Learning
• Bootstrapping LLMs via Preference-Based Policy Optimization
• Offline Clustering of Preference Learning with Active-data Augmentation
• Greedy Sampling Is Provably Efficient for RLHF
• Ask a Strong LLM Judge when Your Reward Model is Uncertain
• Offline and Online KL-Regularized RLHF under Differential Privacy
• Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
• Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• Multiplayer Nash Preference Optimization
• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

Related

• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
• Provable Offline Preference-Based Reinforcement Learning
• Is RLHF More Difficult than Standard RL?
• Reinforcement Learning from Diverse Human Preferences
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback