2025/01/06 by Yan, Zhaoyi, Yiming Zhang, Zhang, Yiming +16 · 3 citations
Business, Management and Accounting · Computer Science · #Business Process Modeling and Analysis #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #FOS: Computer and information sciences #Fusion #Natural Language Processing Techniques #Philosophy #Semantic Web and Ontologies
paper · pdf · doi:10.48550/arxiv.2501.02795
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/01/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce InfiFusion, an efficient training pipeline designed to integrate multiple domain-specialized Large Language Models (LLMs) into a single pivot model, effectively harnessing the strengths of each source model. Traditional fusion methods either merge model parameters directly or rely on knowledge distillation with rigid assumptions, limiting their flexibility and efficiency. InfiFusion overcomes these limitations by enhancing Universal Logit Distillation (ULD) with Top-K selection and Logits Standardization. We propose two fusion strategies: Pairwise Fusion (InfiFusionp), where each source model knowledge is distilled individually into the pivot model followed by merging and Unified Fusion (InfiFusionu), where knowledge from all source models is distilled simultaneously into the pivot model. InfiFusion outperforms the state-of-the-art models, such as Qwen-2.5-14B-Instruct and Phi-4, across 11 widely applied benchmarks covering reasoning, coding, mathematics, and instruction-following tasks. Notably, InfiFusion achieves this superior performance while significantly reduces computational costs, completing full training with only 160 H800 GPU hours compared to the millions typically required for traditional LLM training.