Aggregate, Decompose, and Fine-Tune: A Simple Yet Effective Factor-Tuning Method for Vision Transformer

2023/11/12 by Dongping Chen, Chen, Dongping
Computer Science · Engineering · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #CCD and CMOS Imaging Sensors #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Enhancement Techniques #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2311.06749

openalex publication_date 2023/11/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements have illuminated the efficacy of some tensorization-decomposition Parameter-Efficient Fine-Tuning methods like LoRA and FacT in the context of Vision Transformers (ViT). However, these methods grapple with the challenges of inadequately addressing inner- and cross-layer redundancy. To tackle this issue, we introduce EFfective Factor-Tuning (EFFT), a simple yet effective fine-tuning method. Within the VTAB-1K dataset, our EFFT surpasses all baselines, attaining state-of-the-art performance with a categorical average of 75.9% in top-1 accuracy with only 0.28% of the parameters for full fine-tuning. Considering the simplicity and efficacy of EFFT, it holds the potential to serve as a foundational benchmark. The code and model are now available at https://github.com/Dongping-Chen/EFFT-EFfective-Factor-Tuning.

Related

• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer
• Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers
• Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation
• Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference
• Adaptive Layer Selection for Efficient Vision Transformer Fine-Tuning