Synchronization of mean-field models on the circle

2025/07/30 by Polyanskiy, Yury, Rigollet, Philippe, Yao, Andrew · 4 citations
#Analysis of PDEs (math.AP) #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2507.22857

Abstract

This paper considers a mean-field model of n interacting particles whose state space is the unit circle, a generalization of the classical Kuramoto model. Global synchronization is said to occur if after starting from almost any initial state, all particles coalesce to a common point on the circle. We propose a general synchronization criterion in terms of L₁-norm of the third derivative of the particle interaction function. As an application we resolve a conjecture for the so-called self-attention dynamics (stylized model of transformers), by showing synchronization for all β≥ -0.16, which significantly extends the previous bound of 0≤ β≤ 1 from Criscitiello, Rebjock, McRae, and Boumal (2024). We also show that global synchronization does not occur when β< -2/3.

Citations

• Quantitative Clustering in Mean-Field Transformer Models
• A Unified Perspective on the Dynamics of Deep Transformers
• The random graph process is globally synchronizing
• Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization
• The Asymptotic Behavior of Attention in Transformers
• Clustering in Causal Attention Masking
• Emergence of meta-stable clustering in mean-field transformer models
• Dynamic metastability in the self-attention model
• Synchronization on circles and spheres with nonlinear interactions
• Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
• Setting the Record Straight on Transformer Oversmoothing
• A mathematical perspective on Transformers
• The emergence of clusters in self-attention dynamics
• Expander graphs are globally synchronizing
• Rank Diminishing in Deep Neural Networks
• Revisiting Over-smoothing in BERT from the Perspective of Graph
• Sinkformers: Transformers with Doubly Stochastic Attention
• Long-time dynamics for a simple aggregation equation on the sphere
• Attention Is All You Need
• From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators

Cited by

• Nonlinear diffusion limit of non-local interactions on a sphere
• Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Critical attention scaling in long-context transformers
• A multiscale analysis of mean-field transformers in the moderate interaction regime

Related

• Synchronization on circles and spheres with nonlinear interactions
• Exploring the Interplay of Excitatory and Inhibitory Interactions in the Kuramoto Model on Circle Topologies
• Synchronization on the circle
• Mean field repulsive Kuramoto models: Phase locking and spatial signs
• On the complete phase synchronization for the Kuramoto model in the mean-field limit