2024/06/27 by Shouchang Guo, Guo, Shouchang, Sonam Damani +3 · 1 voice
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Emerging Technologies (cs.ET) #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Natural Language Processing Techniques #Signal Processing (eess.SP) #Speech Recognition and Synthesis #Topic Modeling #cs.AI #cs.CL #cs.ET #cs.LG #eess.SP #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2406.19486
openalex publication_date 2024/06/27 · arxiv published 2024/06/27 · arxiv updated 2024/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs. In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.