Transformer Dissection: A Unified Understanding of Transformer's\n Attention via the Lens of Kernel

2019/08/30 by Yao-Hung Hubert Tsai, Tsai, Yao-Hung Hubert, Shaojie Bai +7 · 10 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1908.11775

openalex publication_date 2019/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer is a powerful architecture that achieves superior performance on\nvarious sequence learning tasks, including neural machine translation, language\nunderstanding, and sequence prediction. At the core of the Transformer is the\nattention mechanism, which concurrently processes all inputs in the streams. In\nthis paper, we present a new formulation of attention via the lens of the\nkernel. To be more precise, we realize that the attention can be seen as\napplying kernel smoother over the inputs with the kernel scores being the\nsimilarities between inputs. This new formulation gives us a better way to\nunderstand individual components of the Transformer's attention, such as the\nbetter way to integrate the positional embedding. Another important advantage\nof our kernel-based formulation is that it paves the way to a larger space of\ncomposing Transformer's attention. As an example, we propose a new variant of\nTransformer's attention which models the input as a product of symmetric\nkernels. This approach achieves competitive performance to the current state of\nthe art model with less computation. In our experiments, we empirically study\ndifferent kernel construction strategies on two widely used tasks: neural\nmachine translation and sequence prediction.\n

Cited by

• Deep learning for pedestrians: backpropagation in Transformers
• A Coulomb Particle Model for Learning Kernel Attention in Transformers
• Integrating Fourier Neural Operator with Diffusion Model for Autoregressive Predictions of Three-dimensional Turbulence
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
• Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
• Translution: Unifying Self-attention and Convolution for Adaptive and Relative Modeling
• Design Principles for Sequence Models via Coefficient Dynamics
• Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations
• Graded Transformers
• Efficient Attention Mechanisms for Large Language Models: A Survey

Related

• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models
• Attention Is All You Need
• Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks
• Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers