sDPO: Don't Use Your Data All at Once

2024/03/28 by Dahyun Kim, Kim, Dahyun, Yungi Kim +11 · 4 citations
Health Professions · Medicine · #Artificial Intelligence (cs.AI) #Cardiovascular Health and Risk Factors #Computation and Language (cs.CL) #FOS: Computer and information sciences #Medical Coding and Health Information

paper · pdf · doi:10.48550/arxiv.2403.19270

openalex publication_date 2024/03/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As development of large language models (LLM) progresses, aligning them with human preferences has become increasingly important. We propose stepwise DPO (sDPO), an extension of the recently popularized direct preference optimization (DPO) for alignment tuning. This approach involves dividing the available preference datasets and utilizing them in a stepwise manner, rather than employing it all at once. We demonstrate that this method facilitates the use of more precisely aligned reference models within the DPO training framework. Furthermore, sDPO trains the final model to be more performant, even outperforming other popular LLMs with more parameters.

Cited by

• GRPO-λ: Credit Assignment improves LLM Reasoning
• MotionFlux: Efficient Text-Guided Motion Generation through Rectified Flow Matching and Preference Alignment
• CTR-Driven Ad Text Generation via Online Feedback Preference Optimization
• Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization

Related

• Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
• Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
• β-DPO: Direct Preference Optimization with Dynamic β
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
• SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment