LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization

2025/07/06 by Xu‐Jia Wang, Wang, Xujia, Yihong Qi +3
Engineering · Materials Science · #3D IC and TSV technologies #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Phase-change materials and chalcogenides #Photonic and Optical Devices

paper · pdf · doi:10.48550/arxiv.2507.04487

openalex publication_date 2025/07/06 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about 27% compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training. The source code is available at https://github.com/KlozeWang/LoSiA.

Citations

• Recurrent Knowledge Identification and Fusion for Language Model Continual Learning
• Low-Rank Adaptation for Foundation Models: A Comprehensive Review
• LoRA vs Full Fine-tuning: An Illusion of Equivalence
• Qwen2.5-Coder Technical Report
• TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation
• KIF: Knowledge Identification and Fusion for Language Model Continual Learning
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• Knowledge Circuits in Pretrained Transformers
• LoRA Learns Less and Forgets Less
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• Asymmetry in Low-Rank Adapters of Foundation Models
• PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
• LoRA+: Efficient Low Rank Adaptation of Large Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• A Closer Look at the Limitations of Instruction Tuning
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• ReLoRA: High-Rank Training Through Low-Rank Updates
• Task-Specific Skill Localization in Fine-tuned Language Models
• PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance
• Training Verifiers to Solve Math Word Problems
• Finetuned Language Models Are Zero-Shot Learners
• Program Synthesis with Large Language Models
• Measuring Massive Multitask Language Understanding
• Aligning AI With Shared Human Values
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Importance Estimation for Neural Network Pruning
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Parameter-Efficient Transfer Learning for NLP
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Related

• HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition
• PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized?