Bootstrapping LLMs via Preference-Based Policy Optimization

2025/11/17 by Jia Chen, Jia, Chen
Computer Science · #Machine Learning and Data Classification #Recommender Systems and Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2511.12867

Abstract

Bootstrapping large language models (LLMs) through preference-based policy optimization offers a promising direction for aligning model behavior with human preferences without relying on extensive manual annotations. In this work, we propose a novel preference-based policy optimization (PbPO) framework that formulates the learning process as a min-max game between the main policy and a reward model (RM). The RM is constrained within a confidence set derived from preference data to ensure reliable exploitation. Our iterative online algorithm actively collects preference data through guided exploration of the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees for our method, establishing high-probability regret bounds for both settings with sequence-level RM and token-level RM, demonstrating its effectiveness in bootstrapping LLMs. Extensive experiments on five benchmarks show that our approach consistently outperforms existing state-of-the-art preference optimization techniques.

Citations

• DPO Meets PPO: Reinforced Token Optimization for RLHF
• Active Preference Optimization for Sample Efficient RLHF
• Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
• Direct Language Model Alignment from Online AI Feedback
• WARM: On the Benefits of Weight Averaged Reward Models
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
• On the Sensitivity of Reward Inference to Misspecified Human Models
• Scaling Laws for Reward Model Overoptimization
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Learning to summarize from human feedback
• Language Models are Few-Shot Learners
• Fine-Tuning Language Models from Human Preferences
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• SALMON: Self-Alignment with Instructable Reward Models
• Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment
• BOND: Aligning LLMs with Best-of-N Distillation
• Bootstrapping Language Models with DPO Implicit Rewards
• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
• Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
• SimPO: Simple Preference Optimization with a Reference-Free Reward

Related

• Mutual-Taught for Co-adapting Policy and Reward Models
• Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
• Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
• Fine-Tuning Language Models with Reward Learning on Policy