LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently

2025/02/03 by Y. T. Zhang, Fanghui Liu, Zhang, Yuanhe +3 · 9 citations
Engineering · Computer Science · #Sparse and Compressive Sensing Techniques #Medical Image Segmentation Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2502.01235

Abstract

This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA, Hu et al. 2022) in large language models as a case study. We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one-step full fine-tuning gradient. This result suggests that, by properly initializing the adapters using the one-step full gradient, subspace alignment can be achieved immediately and applicable to both linear and nonlinear models. Building on our theory, we propose a theory-driven algorithm, LoRA-One, where the linear convergence (as well as generalization) is built and incorporating preconditioners theoretically helps mitigate the effects of ill-conditioning. Besides, our theory reveals connections between LoRA-One and other gradient-alignment-based methods, helping to clarify misconceptions in the design of such algorithms. LoRA-One achieves significant empirical improvements over LoRA and its variants across benchmarks in natural language understanding, mathematical reasoning, and code generation. Code is available at: https://github.com/YuanheZ/LoRA-One.

Cited by

• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
• On the Convergence Rate of LoRA Gradient Descent
• Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
• LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
• Model Merging with Functional Dual Anchors
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
• IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
• Multiple Choice Learning of Low Rank Adapters for Language Modeling

Related

• CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Models
• Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning
• Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
• LoRA² : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models
• Uni-LoRA: One Vector is All You Need