A multiscale analysis of mean-field transformers in the moderate interaction regime

2025/09/29 by Bruno, Giuseppe, Pasqualotto, Federico, Agazzi, Andrea · 6 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Probability (math.PR)

paper · doi:10.48550/arxiv.2509.25040

Abstract

In this paper, we study the evolution of tokens through the depth of encoder-only transformer models at inference time by modeling them as a system of particles interacting in a mean-field way and studying the corresponding dynamics. More specifically, we consider this problem in the moderate interaction regime, where the number N of tokens is large and the inverse temperature parameter β of the model scales together with N. In this regime, the dynamics of the system displays a multiscale behavior: a fast phase, where the token empirical measure collapses on a low-dimensional space, an intermediate phase, where the measure further collapses into clusters, and a slow one, where such clusters sequentially merge into a single one. We provide a rigorous characterization of the limiting dynamics in each of these phases and prove convergence in the above mentioned limit, exemplifying our results with some simulations.

Citations

• Critical attention scaling in long-context transformers
• Attention's forward pass and Frank-Wolfe
• Synchronization of mean-field models on the circle
• Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
• Quantitative Clustering in Mean-Field Transformer Models
• Scalable-Softmax Is Superior for Attention
• A Unified Perspective on the Dynamics of Deep Transformers
• Universal approximations of quasilinear PDEs by finite distinguishable particle systems
• Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization
• Solutions of stationary McKean-Vlasov equation on a high-dimensional sphere and other Riemannian manifolds
• Measure-to-measure interpolation using Transformers
• Clustering in Causal Attention Masking
• Emergence of meta-stable clustering in mean-field transformer models
• Dynamic metastability in the self-attention model
• Clustering in pure-attention hardmax transformers and its role in sentiment analysis
• Synchronization on circles and spheres with nonlinear interactions
• A mathematical perspective on Transformers
• YaRN: Efficient Context Window Extension of Large Language Models
• The emergence of clusters in self-attention dynamics
• Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
• Porous medium equation and cross-diffusion systems as limit of nonlocal interaction
• Sinkformers: Transformers with Doubly Stochastic Attention
• Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Reformer: The Efficient Transformer
• Neural Ordinary Differential Equations
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Attention Is All You Need
• Neural Machine Translation by Jointly Learning to Align and Translate

Cited by

• Nonlinear diffusion limit of non-local interactions on a sphere
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Normalization in Attention Dynamics
• On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers
• Critical attention scaling in long-context transformers
• YuriiFormer: A Suite of Nesterov-Accelerated Transformers

Related

• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• The Mean-Field Dynamics of Transformers
• Quantitative Clustering in Mean-Field Transformer Models
• Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization
• Emergence of meta-stable clustering in mean-field transformer models