A mathematical perspective on Transformers

2023/12/17 by Borjan Geshkovski, Cyril Letrouit, Geshkovski, Borjan +5 · 5 voices · 73 citations
Computer Science · Engineering · Mathematics · #Artificial intelligence #Computer science #Electrical engineering #Engineering #Management science #Perspective (graphical) #Topic Modeling #Transformer #cs.LG #math.AP #math.DS

paper · pdf · doi:10.48550/arxiv.2312.10794

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/12/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Transformers play a central role in the inner workings of large language models. We develop a mathematical framework for analyzing Transformers based on their interpretation as interacting particle systems, which reveals that clusters emerge in long time. Our study explores the underlying theory and offers new perspectives for mathematicians as well as computer scientists.

Cited by

• Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
• On Transformer Dynamics
• On the diffusive-mean field limit of a kinetic weakly interacting particle system
• Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
• The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path
• The Mean-Field Dynamics of Transformers
• What Neuroscience Can Teach AI About Learning in Continuously Changing Environments
• Context-Selective State Space Models: Feedback is All You Need
• Benign nonconvexity of synchronization landscape induced by graph skeletons
• Ground states and phase transitions for an aggregation model with fast diffusion on sphere
• NRGPT: An Energy-based Alternative for GPT
• Optimizing Rank for High-Fidelity Implicit Neural Representations
• Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
• Token Sample Complexity of Attention
• Solving Oversmoothing in GNNs via Nonlocal Message Passing: Algebraic Smoothing and Depth Scalability
• Optimal and Diffusion Transports in Machine Learning
• Nonlinear diffusion limit of non-local interactions on a sphere
• Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
• Global synchronization beyond dense graphs: the case of threshold graphs
• How Particle-System Random Batch Methods Enhance Graph Transformer: Memory Efficiency and Parallel Computing Strategy
• Forget BIT, It is All about TOKEN: Towards Semantic Information Theory for LLMs
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with L₁ Prior
• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers
• Normalization in Attention Dynamics
• On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers
• Turnpike property for hierarchical optimal control problems: from particle systems to hydrodynamic equations
• Formation of clusters and coarsening in weakly interacting diffusions
• Dimension-Free Minimax Rates for Learning Pairwise Interactions in Attention-Style Models
• Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• Critical attention scaling in long-context transformers
• A Mathematical Explanation of Transformers for Large Language Models and GPTs
• Machine Learning and Control: Foundations, Advances, and Perspectives
• Transformers through the lens of support-preserving maps between measures
• A multiscale analysis of mean-field transformers in the moderate interaction regime
• Global energy minimizers for a diffusion-aggregation model on sphere
• Chaos in reason: How chain-of-thought LLMs can look for an answer
• Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness
• Localmax dynamics for attention in transformers and its asymptotic behavior
• Token Homogenization under Positional Bias
• Dynamical Low-Rank Approximations for Kalman Filtering
• Preconditioned Regularized Wasserstein Proximal Sampling
• Wavy Transformer
• Provable Low-Frequency Bias of In-Context Learning of Representations
• Multispin Physics of AI Tipping Points and Hallucinations
• Synchronization of mean-field models on the circle
• Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
• Graded Transformers
• TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations
• YuriiFormer: A Suite of Nesterov-Accelerated Transformers
• Gradient descent avoids strict saddles with a simple line-search method too
• Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics
• A unified framework for establishing the universal approximation of transformer-type architectures
• Finding Clustering Algorithms in the Transformer Architecture
• T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders
• RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
• Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods
• Revisiting Transformers with Insights from Image Filtering and Boosting
• Attention-Only Transformers via Unrolled Subspace Denoising
• Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
• Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
• Attention with Trained Embeddings Provably Selects Important Tokens
• Physical models realizing the transformer architecture of large language models
• Feed-Forward Steering in Transformer Residual Dynamics
• Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency
• The Transformer as a Polar State Estimator
• Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures
• Patnaik-Pearson intrinsic dimension for internal representations of neural networks
• Flowers: A Warp Drive for Neural PDE Solvers
• Learning Enhanced Ensemble Filters
• Quantitative Clustering in Mean-Field Transformer Models
• Diffusion Operator Geometry of Feedforward Representations
• Why do LLMs attend to the first token?

Discussions

• A Mathematical Perspective on Transformers [hn, 82 points, 6 comments]
• Just got done reading arxiv.org/abs/2312.10794 Part One (aka Sections 2,3) provide maybe the nicest and most intuitive descriptions of transformers that I've seen. Simple, easy, direct; no fiddle-fadd [bsky, 1 points, 1 comments]
• very cool paper on math and transformers arxiv.org/abs/2312.10794 [bsky, 0 points, 0 comments]
• This is a nice trick & observation I hadn't seen before: look at depth into the network as "time," and the way a transformer moves (normalized) vectors around can look like an interacting particle sys [bsky, 0 points, 0 comments]
• Have you seen this : arxiv.org/abs/2312.10794 Still one of my favorites on it [bsky, 0 points, 0 comments]

Related

• A mathematical perspective on transformers
• Why transformers are obviously good models of language
• The Mean-Field Dynamics of Transformers
• On Limitations of the Transformer Architecture
• What Formal Languages Can Transformers Express? A Survey