Pay Attention to Small Weights

2025/06/26 by Chao Zhou, Tom Jacobs, Zhou, Chao +5
Health Professions · Business, Management and Accounting · #Obesity and Health Practices #Global Public Health Policies and Epidemiology

paper · pdf · doi:10.48550/arxiv.2506.21374

Abstract

Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free -- the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.

Citations

• Sparsity May Be All You Need: Sparse Random Parameter Adaptation
• Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models
• Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies
• LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
• BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks
• MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
• LoRA Learns Less and Forgets Less
• Mixture of LoRA Experts
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
• Masks, Signs, And Learning Rate Rewinding
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Efficient Large Language Models: A Survey
• MultiLoRA: Democratizing LoRA for Better Multi-Task Learning
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
• Parameter-Efficient Fine-Tuning without Introducing New Latency
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• LLaMA: Open and Efficient Foundation Language Models
• OPT: Open Pre-trained Transformer Language Models
• 8-bit Optimizers via Block-wise Quantization
• LoRA: Low-Rank Adaptation of Large Language Models
• Learning a Single Neuron with Bias Using Gradient Descent
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Language Models are Few-Shot Learners
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Implicit Regularization in Deep Matrix Factorization
• Training Deep Nets with Sublinear Memory Cost
• Deep Residual Learning for Image Recognition
• The Singular Value Decomposition, Applications and Beyond
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Less is More: Selective Layer Finetuning with SubTuning
• Why Fine-grained Labels in Pretraining Benefit Generalization?
• Low-rank Attention Side-Tuning for Parameter-Efficient Fine-Tuning
• Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks