Memory Limitations of Prompt Tuning in Transformers

2025/08/30 by Maxime Meyer, Meyer, Maxime, Mario Michelessa +5
Computer Science · Psychology · #FOS: Computer and information sciences #Language Development and Disorders #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2509.00421

openalex publication_date 2025/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the empirical success of prompt tuning in adapting pretrained language models to new tasks, theoretical analyses of its capabilities remain limited. Existing theoretical work primarily addresses universal approximation properties, demonstrating results comparable to standard weight tuning. In this paper, we explore a different aspect of the theory of transformers: the memorization capability of prompt tuning. We provide two principal theoretical contributions. First, we prove that the amount of information memorized by a transformer cannot scale faster than linearly with the prompt length. Second, and more importantly, we present the first formal proof of a phenomenon empirically observed in large language models: performance degradation in transformers with extended contexts. We rigorously demonstrate that transformers inherently have limited memory, constraining the amount of information they can retain, regardless of the context size. This finding offers a fundamental understanding of the intrinsic limitations of transformer architectures, particularly their ability to handle long sequences.

Citations

• A Theoretical Framework for Prompt Engineering: Approximating Smooth Functions with Transformer Prompts
• Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
• On the Optimal Memorization Capacity of Transformers
• Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models
• Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Protein Multimer Structure Prediction via Prompt Learning
• Prompting a Pretrained Transformer Can Be a Universal Approximator
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
• How Smooth Is Attention?
• LooGLE: Can Long-Context Language Models Understand Long Contexts?
• When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations
• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
• Lost in the Middle: How Language Models Use Long Contexts
• On the Role of Attention in Prompt-tuning
• Memorization Capacity of Multi-Head Attention in Transformers
• Universality and Limitations of Prompt Tuning
• The emergence of clusters in self-attention dynamics
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
• Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery
• MaPLe: Multi-modal Prompt Learning
• Large Language Models are Zero-Shot Reasoners
• PaLM: Scaling Language Modeling with Pathways
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Sinkformers: Transformers with Doubly Stochastic Attention
• Scaling Vision Transformers
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• The Lipschitz Constant of Self-Attention
• Language Models are Few-Shot Learners
• Generating Wikipedia by Summarizing Long Sequences
• Attention Is All You Need
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Universality and Limitations of Prompt Tuning
• Do Transformers Need Deep Long-Range Memory
• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
• Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data
• Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency