Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity

2021/06/30 by Arthur Paul Jacot, Arthur Jacot, François Ged +9 · 10 citations
Computer Science · Mathematics · Physics and Astronomy · #Artificial intelligence #Computer science #Dynamics (music) #FOS: Computer and information sciences #Geometry #Initialization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical optimization #Mathematics #Meteorology #Physics #Quantum chaos and dynamical systems #Saddle #Saddle point #Statistical Mechanics and Entropy #Statistical physics #Symmetry (geometry) #Training (meteorology) #cs.LG #stat.ML #stochastic dynamics and bifurcation

paper · pdf · doi:10.48550/arxiv.2106.15933

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/06/30 · arxiv created 2022/01/31 · arxiv updated 2022/02/01 · openalex created_date 2022/07/24 · openalex updated_date 2026/08/06

Abstract

The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance σ² of the parameters at initialization θ₀. For DLNs of width w, we show a phase transition w.r.t. the scaling γ of the variance σ²=w^-γ as w→∞: for large variance (γ<1), θ₀ is very close to a global minimum but far from any saddle point, and for small variance (γ>1), θ₀ is close to a saddle point and far from any global minimum. While the first case corresponds to the well-studied NTK regime, the second case is less understood. This motivates the study of the case γ→ +∞, where we conjecture a Saddle-to-Saddle dynamics: throughout training, gradient descent visits the neighborhoods of a sequence of saddles, each corresponding to linear maps of increasing rank, until reaching a sparse global minimum. We support this conjecture with a theorem for the dynamics between the first two saddles, as well as some numerical experiments.

Cited by

• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
• Secret mixtures of experts inside your LLM
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• Learning In-context \pmbn-grams with Transformers: Sub-\pmbn-grams Are Near-stationary Points
• Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks
• A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
• The emergence of sparse attention: impact of data distribution and benefits of repetition
• Low-Rank Adaptation Redux for Large Models

Related

• Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
• Saddle-to-Saddle Dynamics in Diagonal Linear Networks
• An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models
• Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability