Parameter-Efficient Tuning with Special Token Adaptation

2022/10/10 by James Y. Huang, Yang, Xiaocong, Wenxuan Zhou +4 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2210.04382

openalex publication_date 2022/10/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to full finetuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models

Cited by

• Targeted Error Correction in Knowledge Distillation: Small Language Models Surpass GPT

Related

• Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning
• How fine can fine-tuning be? Learning efficient language models
• Layer-Wise Evolution of Representations in Fine-Tuned Transformers: Insights from Sparse AutoEncoders
• Parameter-Efficient Fine-Tuning With Adapters
• Parameter-Efficient Fine-Tuning without Introducing New Latency