Understanding the performance gap between online and offline alignment algorithms

2024/05/14 by Yunhao Tang, Tang, Yunhao, Daniel Guo +19 · 13 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gene expression and cancer classification #Genomics and Phylogenetic Studies #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2405.08448

openalex publication_date 2024/05/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.

Cited by

• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
• Humanline: Online Alignment as Perceptual Loss
• General Exploratory Bonus for Optimistic Exploration in RLHF
• Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
• PLaMo 2 Technical Report
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Models
• Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMs

Related

• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
• Solving the Inverse Alignment Problem for Efficient RLHF
• SAIL: Self-Improving Efficient Online Alignment of Large Language Models
• On The Global Convergence Of Online RLHF With Neural Parametrization
• The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback