LoRA vs Full Fine-tuning: An Illusion of Equivalence

2024/10/28 by Reece Shuttleworth, Shuttleworth, Reece, Jacob Andreas +5 · 8 voices · 46 citations
Computer Science · Engineering · Mathematics · Psychology · #Cognitive psychology #Economics #Embedded Systems Design Techniques #Environmental science #Equivalence (formal languages) #Fault Detection and Control Systems #Illusion #Materials science #Mathematics #Parallel Computing and Optimization Techniques #Psychology #Pure mathematics

paper · pdf · doi:10.48550/arxiv.2410.21228

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, are their learned solutions really equivalent? We study how LoRA and full-finetuning change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure: weight matrices trained with LoRA have new, high-ranking singular vectors, which we call intruder dimensions, while those trained with full fine-tuning do not. Further, we extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension -- by causally intervening on the intruder dimensions by changing their associated singular values post-fine-tuning, we show that they cause forgetting. Moreover, scaling them down significantly improves modeling of the pre-training distribution with a minimal drop in downstream task performance. Given this, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. This will be amplified during continual learning because of sequentially fine-tuning, and we show that LoRA models do accumulate intruder dimensions here tend to perform worse in this setting, emphasizing the practicality of our findings.

Cited by

• How Meta-Learning Shapes LoRA Adapter Geometry in Speech Deepfake Detection
• On the Convergence of Stochastic Low-Rank Adaptation
• TopoTuner: Topological Finetuning of Large Language Models
• Hide and Seek in Embedding Space: Geometry-based Steganography and Detection in Large Language Models
• One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
• Position: Modular Memory is the Key to Continual Learning Agents
• Learning to Reason in 13 Parameters
• Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization
• The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
• Selective LoRA for Visual Tokens and Attention Heads
• On the Convergence Rate of LoRA Gradient Descent
• Mitigating Forgetting in Low Rank Adaptation
• Parameter-Efficient Fine-Tuning for HAR: Integrating LoRA and QLoRA into Transformer Models
• How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
• Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency
• LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups
• Random Initialization of Gated Sparse Adapters
• A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
• VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuning
• On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
• An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
• Revisiting Mixout: An Overlooked Path to Robust Finetuning
• Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptation
• Blockwise Hadamard high-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
• Riemannian Optimization for LoRA on the Stiefel Manifold
• HEFT: A Coarse-to-Fine Hierarchy for Enhancing the Efficiency and Accuracy of Language Model Reasoning
• RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs
• IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
• Singular Value Few-shot Adaptation of Vision-Language Models
• PLoRA: Efficient LoRA Hyperparameter Tuning for Large Models
• Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
• Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
• LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
• MemAscend: System Memory Optimization for SSD-Offloaded LLM Fine-Tuning
• Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data
• PiCa: Parameter-Efficient Fine-Tuning with Column Space Projection
• Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards
• Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
• Enhancing AI Face Realism: Cost-Efficient Quality Improvement in Distilled Diffusion Models with a Fully Synthetic Dataset
• Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5
• Latent Personal Memory: Represent personal memory as dynamic soft prompts
• Training Transformers in Cosine Coefficient Space
• AROMA: Autonomous Rank-one Matrix Adaptation
• How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs

Discussions

• LoRA vs. Full Fine-Tuning: An Illusion of Equivalence [hn, 236 points, 53 comments]
• Important paper on LoRA vs full fine-tuning for continual learning: arxiv.org/abs/2410.21228 Although LoRA restricts weight updates to a low-dimensional subspace, which should reduce forgetting and be [bsky, 25 points, 2 comments]
• LoRA vs. Full Fine-Tuning: An Illusion of Equivalence [bsky, 0 points, 0 comments]
• LoRA vs. Full Fine-Tuning: An Illusion of Equivalence https://arxiv.org/abs/2410.21228 https://news.ycombinator.com/item?id=42085665 [bsky, 0 points, 0 comments]
• Breaking News! ✨ "LoRA vs. Full Fine-Tuning: An Illusion of Equivalence" #BreakingNews #News #CurrentEvents Read more: arxiv.org/abs/2410.21228 [bsky, 0 points, 0 comments]
• LoRA vs. Full Fine-Tuning: An Illusion of Equivalence https://arxiv.org/abs/2410.21228 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2410.21228 I have seen this phenomena in practice when tuning a language model on unstructured clinical communication. 🧵 [bsky, 0 points, 2 comments]
• Or here cosine similarity between sorted singular vectors detecting structural shift after fine tuning. [bsky, 0 points, 1 comments]

Related

• LoRA Learns Less and Forgets Less
• ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws
• Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• LoRA Is Slower Than You Think