FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion

2025/04/09 by Zhong, Longguang, Fanqi Wan, Ziyi Yang +8 · 1 citation
Computer Science · #Advanced Multi-Objective Optimization Algorithms #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2504.06562

openalex publication_date 2025/04/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches often rely solely on selecting the best output for each prompt from source models, which underutilizes their full potential due to limited source knowledge and results in sparse optimization signals. To address this limitation, we propose FuseRL, a novel two-stage framework comprising FuseSFT and FusePO to maximize the utilization of source LLMs. FuseSFT establishes a robust initialization by integrating the strengths of heterogeneous source models through weighted supervised fine-tuning (SFT) on diverse outputs for each prompt. FusePO optimizes weighted preferences based on the outputs of multiple source models to enable superior alignment performance. Extensive experiments demonstrate the effectiveness of our framework across various preference alignment methods, including RLOO, DPO, and SimPO. Using Llama-3.1-8B-Instruct as the target model, our approach achieves state-of-the-art performance among 8B LLMs on the AlpacaEval-2 and Arena-Hard benchmarks. Further analysis suggests that FuseSFT regularizes the training process to reduce overfitting, while FusePO introduces dense and diverse signals for preference optimization.

Citations

• Qwen2.5 Technical Report
• Weighted-Reward Preference Optimization for Implicit Model Fusion
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• FuseChat: Knowledge Fusion of Chat Models
• Knowledge Fusion of Chat LLMs: A Preliminary Technical Report
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• Mixture-of-Agents Enhances Large Language Model Capabilities
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Bridging the Gap between Different Vocabularies for LLM Ensemble
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
• Evolutionary optimization of model merging recipes
• Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
• ORPO: Monolithic Preference Optimization without Reference Model
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Knowledge Fusion of Large Language Models
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark
• Statistical Rejection Sampling Improves Preference Optimization
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• GPT-4 Technical Report
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Measuring Massive Multitask Language Understanding
• Ensemble approach for natural language question answering problem
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Proximal Policy Optimization Algorithms
• Crowdsourcing Multiple Choice Science Questions
• Deep reinforcement learning from human preferences
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Cited by

• Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration

Related

• FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion
• Weighted-Reward Preference Optimization for Implicit Model Fusion
• InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
• Less is More: Improving LLM Alignment via Preference Data Selection
• Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model