Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel

2019/08/30 by Yao-Hung Hubert Tsai, Tsai, Yao-Hung Hubert, Shaojie Bai +8 · 31 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1908.11775

EMNLP 2019

openalex publication_date 2019/08/30 · arxiv created 2019/11/11 · arxiv updated 2019/11/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this paper, we present a new formulation of attention via the lens of the kernel. To be more precise, we realize that the attention can be seen as applying kernel smoother over the inputs with the kernel scores being the similarities between inputs. This new formulation gives us a better way to understand individual components of the Transformer's attention, such as the better way to integrate the positional embedding. Another important advantage of our kernel-based formulation is that it paves the way to a larger space of composing Transformer's attention. As an example, we propose a new variant of Transformer's attention which models the input as a product of symmetric kernels. This approach achieves competitive performance to the current state of the art model with less computation. In our experiments, we empirically study different kernel construction strategies on two widely used tasks: neural machine translation and sequence prediction.

Cited by

• Deep learning for pedestrians: backpropagation in Transformers
• A Coulomb Particle Model for Learning Kernel Attention in Transformers
• Integrating Fourier Neural Operator with Diffusion Model for Autoregressive Predictions of Three-dimensional Turbulence
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
• Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation
• Translution: Unifying Self-attention and Convolution for Adaptive and Relative Modeling
• Design Principles for Sequence Models via Coefficient Dynamics
• Decoupled-Value Attention for Prior-Data Fitted Networks: GP Inference for Physical Equations
• Graded Transformers
• Efficient Attention Mechanisms for Large Language Models: A Survey
• PGKET: A Photonic Gaussian Kernel Enhanced Transformer
• Attention (as Discrete-Time Markov) Chains
• Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs
• SAS: Simulated Attention Score
• Relational inductive biases on attention mechanisms
• A unified framework for establishing the universal approximation of transformer-type architectures
• Finding Clustering Algorithms in the Transformer Architecture
• Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods
• Revisiting Transformers with Insights from Image Filtering and Boosting
• MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
• Attention-Only Transformers via Unrolled Subspace Denoising
• Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
• LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding
• SchoenbAt: Rethinking Attention with Polynomial basis
• Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform
• Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning
• A Reproduction Study: The Kernel PCA Interpretation of Self-Attention Fails Under Scrutiny
• The Transformer as a Polar State Estimator
• Sensoformer: Robust Sim-to-Real Inference on Variable-Geometry Sensor Sets via Physics-Structured Randomization
• Ordinary Least Squares as an Attention Mechanism
• Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation

Related

• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models
• Attention Is All You Need
• Systematic Generalization and Emergent Structures in Transformers Trained on Structured Tasks
• Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers