Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

2025/06/15 by Bo Wang, Wang, Bo, Runyu Peng +16 · 10 citations
Economics, Econometrics and Finance · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Diverse Scientific and Economic Studies #FOS: Computer and information sciences #Law, Economics, and Judicial Systems #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.00018

openalex publication_date 2025/06/15 · openalex created_date 2025/10/20 · openalex updated_date 2026/07/28

Abstract

Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to 25% relative gain and 6% absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.

Cited by

• Rethinking Expert Trajectory Utilization in LLM Post-training
• Bridging VLMs and Embodied Intelligence with Deliberate Practice Policy Optimization
• Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
• RealDPO: Real or Not Real, that is the Preference
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
• One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
• SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• Sample-efficient LLM Optimization with Reset Replay
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

Related

• Explicit Preference Optimization: No Need for an Implicit Reward Model
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Minor DPO reject penalty to increase training robustness
• Robust Preference Optimization through Reward Model Distillation
• A Principled Loss Function for Direct Language Model Alignment