Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

2024/02/01 by E Weinan, Wang, Mingze, E, Weinan · 10 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2402.00522

openalex publication_date 2024/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We conduct a systematic study of the approximation properties of Transformer for sequence modeling with long, sparse and complicated memory. We investigate the mechanisms through which different components of Transformer, such as the dot-product self-attention, positional encoding and feed-forward layer, affect its expressive power, and we study their combined effects through establishing explicit approximation rates. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads. These theoretical insights are validated experimentally and offer natural suggestions for alternative architectures.

Cited by

• The Effect of Attention Head Count on Transformer Approximation
• Allocation of Parameters in Transformers
• Transformers through the lens of support-preserving maps between measures
• Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
• A unified framework for establishing the universal approximation of transformer-type architectures
• On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
• Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
• Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective
• Approximation Bounds for Transformer Networks with Application to Regression

Related

• The Effect of Attention Head Count on Transformer Approximation
• Introduction to Sequence Modeling with Transformers
• Approximation Rate of the Transformer Architecture for Sequence Modeling
• Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
• Transformer Quality in Linear Time