Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization

2023/05/06 by Anastasia Razdaibiedina, Razdaibiedina, Anastasia, Yuning Mao +10 · 15 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.03937

openalex publication_date 2023/05/06 · openalex created_date 2023/05/10 · openalex updated_date 2026/07/28

Abstract

Prompt tuning is one of the successful approaches for parameter-efficient tuning of pre-trained language models. Despite being arguably the most parameter-efficient (tuned soft prompts constitute <0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters. In this work, we introduce Residual Prompt Tuning - a simple and efficient method that significantly improves the performance and stability of prompt tuning. We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection. Our experiments show that Residual Prompt Tuning significantly outperforms prompt tuning on SuperGLUE benchmark. Notably, our method reaches +7 points improvement over prompt tuning with T5-Base and allows to reduce the prompt length by 10x without hurting performance. In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings.

Cited by

• Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
• NAS-LoRA: Empowering Parameter-Efficient Fine-Tuning for Visual Foundation Models with Searchable Adaptation
• LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
• Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
• Restoring Pruned Large Language Models via Lost Component Compensation
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
• DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
• Cross-Prompt Encoder for Low-Performing Languages
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity
• A Survey on Prompt Tuning
• Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR
• Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning
• Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

Related

• SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings
• Structured Prompt Tuning
• The Power of Scale for Parameter-Efficient Prompt Tuning
• FPT: Improving Prompt Tuning Efficiency via Progressive Training
• Decomposed Prompt Tuning via Low-Rank Reparameterization