Ahead-of-Time P-Tuning

2023/05/18 by Daniil Gavrilov, Gavrilov, Daniil, Nikita Balagansky +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.10835

openalex publication_date 2023/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose Ahead-of-Time (AoT) P-Tuning, a novel parameter-efficient fine-tuning method for pre-trained Language Models (LMs) that adds input-dependent bias before each Transformer layer. We evaluate AoT P-Tuning on GLUE and SuperGLUE benchmarking datasets using RoBERTa and DeBERTa models, showing that it outperforms BitFit and is comparable or better than other baseline methods for efficient fine-tuning. Additionally, we assess the inference overhead of AoT P-Tuning and demonstrate that it introduces negligible overhead compared to established baseline methods. Our method enables multi-task inference with a single backbone LM, making it a practical solution for real-world applications.

Related

• Prototypical Fine-tuning: Towards Robust Performance Under Varying Data Sizes
• Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models
• Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach