Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity

2021/06/30 by Arthur Paul Jacot, François Gaston Ged, Jacot, Arthur +7 · 6 citations
Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Quantum chaos and dynamical systems #Statistical Mechanics and Entropy #stochastic dynamics and bifurcation

paper · pdf · doi:10.48550/arxiv.2106.15933

openalex publication_date 2021/06/30 · openalex created_date 2022/07/24 · openalex updated_date 2026/07/28

Abstract

The dynamics of Deep Linear Networks (DLNs) is dramatically affected by the variance σ² of the parameters at initialization θ₀. For DLNs of width w, we show a phase transition w.r.t. the scaling γ of the variance σ²=w^-γ as w→∞: for large variance (γ<1), θ₀ is very close to a global minimum but far from any saddle point, and for small variance (γ>1), θ₀ is close to a saddle point and far from any global minimum. While the first case corresponds to the well-studied NTK regime, the second case is less understood. This motivates the study of the case γ→ +∞, where we conjecture a Saddle-to-Saddle dynamics: throughout training, gradient descent visits the neighborhoods of a sequence of saddles, each corresponding to linear maps of increasing rank, until reaching a sparse global minimum. We support this conjecture with a theorem for the dynamics between the first two saddles, as well as some numerical experiments.

Cited by

• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
• Secret mixtures of experts inside your LLM
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• Learning In-context \pmbn-grams with Transformers: Sub-\pmbn-grams Are Near-stationary Points

Related

• Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escape
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
• Saddle-to-Saddle Dynamics in Diagonal Linear Networks
• An Analytical Characterization of Sloppiness in Neural Networks: Insights from Linear Models
• Learning Dynamics of Deep Linear Networks Beyond the Edge of Stability