Prompting a Pretrained Transformer Can Be a Universal Approximator

2024/02/22 by Aleksandar Petrov, Philip H. S. Torr, Petrov, Aleksandar +3 · 10 citations
Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Digital Filter Design and Implementation #FOS: Computer and information sciences #FOS: Mathematics #Functional Analysis (math.FA) #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2402.14753

openalex publication_date 2024/02/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the widespread adoption of prompting, prompt tuning and prefix-tuning of transformer models, our theoretical understanding of these fine-tuning methods remains limited. A key question is whether one can arbitrarily modify the behavior of pretrained model by prompting or prefix-tuning it. Formally, whether prompting and prefix-tuning a pretrained model can universally approximate sequence-to-sequence functions. This paper answers in the affirmative and demonstrates that much smaller pretrained models than previously thought can be universal approximators when prefixed. In fact, the attention mechanism is uniquely suited for universal approximation with prefix-tuning a single attention head being sufficient to approximate any continuous function. Moreover, any sequence-to-sequence function can be approximated by prefixing a transformer with depth linear in the sequence length. Beyond these density-type results, we also offer Jackson-type bounds on the length of the prefix needed to approximate a function to a desired precision.

Cited by

• On the Universality of Transformer Architectures; How Much Attention Is Enough?
• Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding
• Memory Limitations of Prompt Tuning in Transformers
• Urban AI Governance Must Embed Legal Reasonableness for Democratic and Sustainable Cities
• A unified framework for establishing the universal approximation of transformer-type architectures
• Sample Complexity and Representation Ability of Test-time Scaling Paradigms
• Learning to Translate from Soft to Hard LLM Prompts
• Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective
• Approximation Bounds for Transformer Networks with Application to Regression
• Long Context In-Context Compression by Getting to the Gist of Gisting

Related

• Universality and Limitations of Prompt Tuning
• The Effect of Attention Head Count on Transformer Approximation
• Universal Approximation Theorem for a Single-Layer Transformer
• Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?