Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorization

2025/08/28 by Min, Hancheng, Vidal, René · 1 citation
#FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Systems and Control (eess.SY) #electronic engineering #information engineering

paper · doi:10.48550/arxiv.2508.20344

Abstract

Many theoretical studies on neural networks attribute their excellent empirical performance to the implicit bias or regularization induced by first-order optimization algorithms when training networks under certain initialization assumptions. One example is the incremental learning phenomenon in gradient flow (GF) on an overparamerterized matrix factorization problem with small initialization: GF learns a target matrix by sequentially learning its singular values in decreasing order of magnitude over time. In this paper, we develop a quantitative understanding of this incremental learning behavior for GF on the symmetric matrix factorization problem, using its closed-form solution obtained by solving a Riccati-like matrix differential equation. We show that incremental learning emerges from some time-scale separation among dynamics corresponding to learning different components in the target matrix. By decreasing the initialization scale, these time-scale separations become more prominent, allowing one to find low-rank approximations of the target matrix. Lastly, we discuss the possible avenues for extending this analysis to asymmetric matrix factorization problems.

Citations

• Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
• Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation
• Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization
• Transformers learn through gradual rank increase
• Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
• On the Implicit Bias in Deep-Learning Algorithms
• Get rid of your constraints and reparametrize: A study in NNLS and implicit bias
• Implicit Sparse Regularization: The Impact of Depth and Early Stopping
• Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Implicit Regularization in Deep Matrix Factorization
• Implicit Regularization of Discrete Gradient Dynamics in Linear Neural\n Networks
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• Attention Is All You Need
• Implicit Regularization in Matrix Factorization
• Deep Residual Learning for Image Recognition
• ImageNet classification with deep convolutional neural networks
• Local Minima and Convergence in Low-Rank Semidefinite Programming

Cited by

• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data

Related

• Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing
• Stability properties of gradient flow dynamics for the symmetric low-rank matrix factorization problem
• Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank