2025/01/31 by Shengyang Sun, Sun, Shengyang, Yian Zhang +27 · 1 voice · 2 citations
Computer Science · #Advanced Database Systems and Queries #Computation and Language (cs.CL) #Data Management and Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2502.00203
openalex publication_date 2025/01/31 · arxiv published 2025/01/31 · arxiv updated 2025/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment.