LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

2024/02/28 by Yichao Wu, Wu, Yichao, Yafei Xiang +7
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.08822

openalex publication_date 2024/02/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In addressing the computational and memory demands of fine-tuning Large Language Models(LLMs), we propose LoRA-SP(Streamlined Partial Parameter Adaptation), a novel approach utilizing randomized half-selective parameter freezing within the Low-Rank Adaptation(LoRA)framework. This method efficiently balances pre-trained knowledge retention and adaptability for task-specific optimizations. Through a randomized mechanism, LoRA-SP determines which parameters to update or freeze, significantly reducing computational and memory requirements without compromising model performance. We evaluated LoRA-SP across several benchmark NLP tasks, demonstrating its ability to achieve competitive performance with substantially lower resource consumption compared to traditional full-parameter fine-tuning and other parameter-efficient techniques. LoRA-SP innovative approach not only facilitates the deployment of advanced NLP models in resource-limited settings but also opens new research avenues into effective and efficient model adaptation strategies.

Related

• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
• Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models
• LoRA Is Slower Than You Think
• LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters