Efficient Low-rank Backpropagation for Vision Transformer Adaptation

2023/09/26 by Yuedong Yang, Yang, Yuedong, Hung-Yueh Chiang +7 · 4 citations
Computer Science · Engineering · #Advanced Image Fusion Techniques #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2309.15275

openalex publication_date 2023/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The increasing scale of vision transformers (ViT) has made the efficient fine-tuning of these large models for specific needs a significant challenge in various applications. This issue originates from the computationally demanding matrix multiplications required during the backpropagation process through linear layers in ViT. In this paper, we tackle this problem by proposing a new Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method. Intuitively, LBP-WHT projects the gradient into a low-rank space and carries out backpropagation. This approach substantially reduces the computation needed for adapting ViT, as matrix multiplication in the low-rank space is far less resource-intensive. We conduct extensive experiments with different models (ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the effectiveness of our method. For instance, when adapting an EfficientFormer-L1 model on CIFAR100, our LBP-WHT achieves 10.4% higher accuracy than the state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the first work to accelerate ViT adaptation with low-rank backpropagation, our LBP-WHT method is complementary to many prior efforts and can be combined with them for better performance.

Cited by

• Efficient Resource-Constrained Training of Transformers via Subspace Optimization
• MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression
• Gradient Inversion Attacks on Parameter-Efficient Fine-Tuning
• SAM4EM: Efficient memory-based two stage prompt-free segment anything model adapter for complex 3D neuroscience electron microscopy stacks

Related

• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
• Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy
• Serial Low-rank Adaptation of Vision Transformer
• Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach
• Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing