Res-Attn : An Enhanced Res-Tuning Approach with Lightweight Attention Mechanism

2023/12/28 by Chaojie Mao, Mao, Chaojie, Zeyinzi Jiang +1
Computer Science · #Machine Learning and ELM #Domain Adaptation and Few-Shot Learning #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2312.16916

Abstract

Res-Tuning introduces a flexible and efficient paradigm for model tuning, showing that tuners decoupled from the backbone network can achieve performance comparable to traditional methods. Existing methods commonly construct the tuner as a set of trainable low-rank decomposition matrices, positing that a low-rank subspace suffices for adapting pre-trained foundational models to new scenarios. In this work, we present an advanced, efficient tuner augmented with low-rank attention, termed Res-Attn , which also adheres to the Res-Tuning framework. Res-Attn utilizes a parallel multi-head attention module equipped with low-rank projections for query, key, and value to execute streamlined attention operations. Through training this lightweight attention module, Res-Attn facilitates adaptation to new scenarios. Our extensive experiments across a range of discriminative and generative tasks showcase the superior performance of our method when compared to existing alternatives

Related

• Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
• Low-rank Attention Side-Tuning for Parameter-Efficient Fine-Tuning
• Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models
• Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modeling
• Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters