Weight Spectra Induced Efficient Model Adaptation

2025/05/29 by Chongjie Si, Si, Chongjie, Mu-Qing Liu +12 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.23099

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-scale foundation models have demonstrated remarkable versatility across a wide range of downstream tasks. However, fully fine-tuning these models incurs prohibitive computational costs, motivating the development of Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA, which introduces low-rank updates to pre-trained weights. Despite their empirical success, the underlying mechanisms by which PEFT modifies model parameters remain underexplored. In this work, we present a systematic investigation into the structural changes of weight matrices during fully fine-tuning. Through singular value decomposition (SVD), we reveal that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact, suggesting that task-specific knowledge is injected into a low-dimensional subspace. Furthermore, we find that the dominant singular vectors are reoriented in task-specific directions, whereas the non-dominant subspace remains stable. Building on these insights, we propose a novel method that leverages learnable rescaling of top singular directions, enabling precise modulation of the most influential components without disrupting the global structure. Our approach achieves consistent improvements over strong baselines across multiple tasks, highlighting the efficacy of structurally informed fine-tuning.

Citations

• See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition
• Mixture-of-Subspaces in Low-Rank Adaptation
• MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
• TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation
• LoRA Learns Less and Forgets Less
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
• Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
• Micrograph segmentations for DDEVD
• Segment Anything
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
• Convolutional Bypasses Are Better Vision Transformer Adapters
• Neural Prompt Search
• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Visual Prompt Tuning
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• Towards a Unified View of Parameter-Efficient Transfer Learning
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• Gradient Descent on Neural Networks Typically Occurs at the Edge of\n Stability
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Parameter-Efficient Transfer Learning for NLP
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Neural Network Acceptability Judgments
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning
• Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Inductive Bias and Spectral Properties of Single-Head Attention in High Dimensions

Related

• DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
• SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
• See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition
• Parameter-Efficient Fine-Tuning via Circular Convolution
• Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning