The Mean-Field Dynamics of Transformers

2025/12/01 by Philippe Rigollet, Rigollet, Philippe · 3 voices · 4 citations
#cs.LG #math-ph #math.DS #math.PR

paper · pdf · doi:10.48550/arxiv.2512.01868

Abstract

We develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.

Citations

• A Unified Perspective on the Dynamics of Deep Transformers
• A mathematical perspective on Transformers
• LoRA: Low-Rank Adaptation of Large Language Models
• Neural ODE Control for Classification, Approximation, and Transport
• Neural Ordinary Differential Equations
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
• Many-body Tipping Dynamics of ChatGPT-like AIs
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Critical attention scaling in long-context transformers
• YuriiFormer: A Suite of Nesterov-Accelerated Transformers

Discussions

• Read last night. Very nice. arxiv.org/abs/2512.01868 [bsky, 21 points, 2 comments]
• do you mean in detail or analyzing it as a system of interacting particles? arxiv.org/abs/2512.018... [bsky, 5 points, 1 comments]
• The arXiv paper uploaded a month ago looked really interesting arxiv.org/abs/2512.01868 [bsky, 0 points, 0 comments]

Related

• Quantitative Clustering in Mean-Field Transformer Models
• A Unified Perspective on the Dynamics of Deep Transformers
• The Asymptotic Behavior of Attention in Transformers
• The emergence of clusters in self-attention dynamics
• A multiscale analysis of mean-field transformers in the moderate interaction regime