On the Universality of Transformer Architectures; How Much Attention Is Enough?

2025/12/20 by Amirreza Abbasi, Abbasi, Amirreza, Mohsen Hooshmand +1
Computer Science · Engineering · #Advanced Neural Network Applications #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · doi:10.48550/arxiv.2512.18445

openalex publication_date 2025/12/20 · openalex created_date 2025/12/24 · openalex updated_date 2026/07/28

Abstract

Transformers are crucial across many AI fields, such as large language models, computer vision, and reinforcement learning. This prominence stems from the architecture's perceived universality and scalability compared to alternatives. This work examines the problem of universality in Transformers, reviews recent progress, including architectural refinements such as structural minimality and approximation rates, and surveys state-of-the-art advances that inform both theoretical and practical understanding. Our aim is to clarify what is currently known about Transformers expressiveness, separate robust guarantees from fragile ones, and identify key directions for future theoretical research.

Citations

• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Universal Approximation with Softmax Attention
• Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective
• Transformers are Expressive, But Are They Expressive Enough for Regression?
• Prompting a Pretrained Transformer Can Be a Universal Approximator
• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
• Sumformer: Universal Approximation for Efficient Transformers
• Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input
• Approximation Rate of the Transformer Architecture for Sequence Modeling
• Your Transformer May Not be as Powerful as You Expect
• Rethinking Attention with Performers
• On Representing (Anti)Symmetric Functions
• O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Linformer: Self-Attention with Linear Complexity
• Are Transformers universal approximators of sequence-to-sequence functions?
• Self-Attention with Relative Position Representations
• Attention Is All You Need

Related

• An Introduction to Transformers
• Transformers are Expressive, But Are They Expressive Enough for Regression?
• Transformers are Universal Predictors
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• A unified framework for establishing the universal approximation of transformer-type architectures