Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance

2025/08/29 by Yao Wang, Wang, Yao, Di Liang +3 · 1 voice · 4 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2508.21741

openalex publication_date 2025/08/29 · arxiv published 2025/08/29 · arxiv updated 2025/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Supervised fine-tuning (SFT) is a pivotal approach to adapting large language models (LLMs) for downstream tasks; however, performance often suffers from the ``seesaw phenomenon'', where indiscriminate parameter updates yield progress on certain tasks at the expense of others. To address this challenge, we propose a novel Core Parameter Isolation Fine-Tuning (CPI-FT) framework. Specifically, we first independently fine-tune the LLM on each task to identify its core parameter regions by quantifying parameter update magnitudes. Tasks with similar core regions are then grouped based on region overlap, forming clusters for joint modeling. We further introduce a parameter fusion technique: for each task, core parameters from its individually fine-tuned model are directly transplanted into a unified backbone, while non-core parameters from different tasks are smoothly integrated via Spherical Linear Interpolation (SLERP), mitigating destructive interference. A lightweight, pipelined SFT training phase using mixed-task data is subsequently employed, while freezing core regions from prior tasks to prevent catastrophic forgetting. Extensive experiments on multiple public benchmarks demonstrate that our approach significantly alleviates task interference and forgetting, consistently outperforming vanilla multi-task and multi-stage fine-tuning baselines.

Citations

• TableBench: A Comprehensive and Complex Benchmark for Table Question Answering
• Qwen2 Technical Report
• Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
• Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inference
• Arcee's MergeKit: A Toolkit for Merging Large Language Models
• Question Calibration and Multi-Hop Modeling for Temporal Question Answering
• Continual Learning for Large Language Models: A Survey
• Unleashing Potential of Evidence in Knowledge-Intensive Dialogue Generation
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusion
• What Makes Pre-trained Language Models Better Zero-shot Learners?
• Searching for Optimal Subword Tokenization in Cross-domain NER
• Training language models to follow instructions with human feedback
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Finetuned Language Models Are Zero-Shot Learners
• LoRA: Low-Rank Adaptation of Large Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
• Language Models are Few-Shot Learners
• Gradient Surgery for Multi-Task Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Parameter-Efficient Transfer Learning for NLP
• Experience Replay for Continual Learning
• Taskonomy: Disentangling Task Transfer Learning
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Gradient Episodic Memory for Continual Learning
• Learning without Forgetting
• Adam: A Method for Stochastic Optimization
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHF
• Tag-Enriched Multi-Attention with Large Language Models for Cross-Domain Sequential Recommendation
• Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
• StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matching

Discussions

• And this paper: arxiv.org/abs/2508.21741 (Not many ablations) [bsky, 0 points, 0 comments]

Related

• Parameter-Efficient Fine-Tuning without Introducing New Latency
• Improved Supervised Fine-Tuning for Large Language Models to Mitigate Catastrophic Forgetting
• Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach
• Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
• HFT: Half Fine-Tuning for Large Language Models