LoKI: Low-damage Knowledge Implanting of Large Language Models

2025/05/28 by Wang, Runyu, Ping, Peng, Guo, Zhengyu +4
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.22120

Abstract

Fine-tuning adapts pretrained models for specific tasks but poses the risk of catastrophic forgetting (CF), where critical knowledge from pretraining is overwritten. To address the issue of CF in a general-purpose framework, we propose Low-damage Knowledge Implanting (LoKI), a parameter-efficient fine-tuning (PEFT) technique that utilizes recent mechanistic understanding of how knowledge is stored in transformer architectures. We compare LoKI against state-of-the-art PEFT methods in two real-world fine-tuning scenarios. The results show that LoKI demonstrates significantly better preservation of general capabilities. At the same time, its task-specific performance is comparable to or even surpasses that of full parameter fine-tuning and these PEFT methods across various model architectures. Our work bridges the mechanistic insights of LLMs' knowledge storage with practical fine-tuning objectives, enabling an effective balance between task-specific adaptation and the retention of general-purpose capabilities.

Related

• Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks
• Pretraining with hierarchical memories: separating long-tail and common knowledge
• Scaling Laws for Forgetting When Fine-Tuning Large Language Models
• SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models
• MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning