2024/04/23 by Yifeng Ding, Ding, Yifeng, Jiawei Liu +7 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Code (set theory) #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Machine Learning and Data Classification #Music and Audio Processing #Physics #Power (physics) #Programming language #Software Engineering (cs.SE)
paper · pdf · doi:10.48550/arxiv.2404.15247
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/04/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft.