Regularization Implies balancedness in the deep linear network

2025/11/03 by Lindsey, Kathryn, Menon, Govind
#14L24 #37C10 #37J15 #37N40 #49J15 #53D20 #68T07 #93B10 #Algebraic Geometry (math.AG) #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2511.01137

Abstract

We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the L² regularizer is minimized on the balanced manifold. This allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. We show that the regularizing flow is exactly solvable using the moment map. This approach provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of model reduction and Bayesian principles.

Citations

• Entropic Regularization in the Deep Linear Network
• An entropy formula for the Deep Linear Network
• Symmetry in Neural Network Parameter Spaces
• Geometry of fibers of the multiplication map of deep linear neural networks
• The geometry of the deep linear network
• On Functional Dimension and Persistent Pseudodimension
• Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
• The Geometry of the Set of Equivalent Linear Neural Networks
• Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
• Abide by the Law and Follow the Flow: Conservation Laws for Gradient\n Flows
• Flat minima generalize for low-rank matrix recovery
• On the Implicit Bias of Gradient Descent for Temporal Extrapolation
• Efficiently Modeling Long Sequences with Structured State Spaces
• Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning\n Dynamics
• Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers
• Gradient descent aligns the layers of deep linear networks
• The Harish-Chandra integral: An introduction with examples
• Algorithmic Regularization in Learning Deep Homogeneous Models: Layers\n are Automatically Balanced
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

Cited by

• An entropy formula for the Deep Linear Network

Related

• On regularization of gradient descent, layer imbalance and flat minima
• Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit
• On the training dynamics of deep networks with L₂ regularization
• Entropic Regularization in the Deep Linear Network
• Manifold Regularization for Locally Stable Deep Neural Networks