Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

2024/10/01 by Yiming Ju, Ju, Yiming, Ziyi Ni +9
Computer Science · Materials Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Magnetic Properties and Applications #Magnetic confinement fusion research #VLSI and Analog Circuit Testing

paper · pdf · doi:10.48550/arxiv.2410.03743

openalex publication_date 2024/10/01 · openalex created_date 2024/11/01 · openalex updated_date 2026/07/28

Abstract

Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to significant training imbalances, potentially resulting in performance degradation. Consequently, we propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders, thereby enhancing the overall effectiveness of SFT. Additionally, we introduce a novel technique, "parameter-selection merging," which outperforms traditional weighted-average methods on five datasets. Further, through analysis and ablation studies, we validate the effectiveness of our method and identify the sources of performance improvements.

Related

• Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models
• DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning
• HFT: Half Fine-Tuning for Large Language Models