Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

2024/07/16 by Yuzhu Mao, Siqi Ping, Mao, Yuzhu +7
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2407.12074

openalex publication_date 2024/07/16 · openalex created_date 2024/09/09 · openalex updated_date 2026/07/28

Abstract

Large pre-trained models, such as large language models (LLMs), present significant resource challenges for fine-tuning due to their extensive parameter sizes, especially for applications in mobile systems. To address this, Low-Rank Adaptation (LoRA) has been developed to reduce resource consumption while maintaining satisfactory fine-tuning results. Despite its effectiveness, the original LoRA method faces challenges of suboptimal performance and overfitting. This paper investigates the intrinsic dimension of the matrix updates approximated by the LoRA method and reveals the performance benefits of increasing this intrinsic dimension. By employing regularization and a gradient masking method that encourages higher intrinsic dimension, the proposed method, termed Regularized and Masked LoRA (RM-LoRA), achieves superior generalization performance with the same or lower trainable parameter budget compared to the original LoRA and its latest variants across various open-source vision and language datasets.

Related

• Less is More: Resource-Efficient Low-Rank Adaptation
• LoRA Is Slower Than You Think
• LoRA² : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models
• GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning
• QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models