β-DPO: Direct Preference Optimization with Dynamic β

2024/07/11 by Junkang Wu, Wu, Junkang, Yuexiang Xie +13 · 23 citations
Computer Science · #Constraint Satisfaction and Optimization #Data Management and Algorithms #Rough Sets and Fuzzy Logic

paper · pdf · doi:10.48550/arxiv.2407.08639

Abstract

Direct Preference Optimization (DPO) has emerged as a compelling approach for training Large Language Models (LLMs) to adhere to human preferences. However, the performance of DPO is sensitive to the fine-tuning of its trade-off parameter β, as well as to the quality of the preference data. We analyze the impact of β and data quality on DPO, uncovering that optimal β values vary with the informativeness of pairwise data. Addressing the limitations of static β values, we introduce a novel framework that dynamically calibrates β at the batch level, informed by data quality considerations. Additionally, our method incorporates β-guided data filtering to safeguard against the influence of outliers. Through empirical evaluation, we demonstrate that our dynamic β adjustment technique significantly improves DPO's performance across a range of models and datasets, offering a more robust and adaptable training paradigm for aligning LLMs with human feedback. The code is available at \urlhttps://github.com/junkangwu/beta-DPO.

Cited by

• Multi-dimensional Preference Alignment by Conditioning Reward Itself
• Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
• TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
• On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenders
• Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
• Abductive Preference Learning
• CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards
• Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
• GRPO-λ: Credit Assignment improves LLM Reasoning
• Fading to Grow: Growing Preference Ratios via Preference Fading Discrete Diffusion for Recommendation
• Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
• Humanline: Online Alignment as Perceptual Loss
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• Adaptive Margin RLHF via Preference over Preferences
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
• Towards Transparent and Incentive-Compatible Collaboration in Decentralized LLM Multi-Agent Systems: A Blockchain-Driven Approach
• What Matters in Data for DPO?
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
• On Negative-aware Preference Optimization for Recommendation
• SGPO: Self-Generated Preference Optimization based on Self-Improver
• Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning

Related

• What Matters in Data for DPO?
• sDPO: Don't Use Your Data All at Once
• Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
• Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
• Minor DPO reject penalty to increase training robustness