Gradient descent aligns the layers of deep linear networks

2018/10/04 by Ziwei Ji, Matus Telgarsky, Ji, Ziwei +1 · 16 citations
Engineering · Computer Science · Medicine · #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Advanced Neuroimaging Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1810.02032

Abstract

This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar results for gradient descent with particular decreasing step sizes): (i) the risk converges to 0; (ii) the normalized i-th weight matrix asymptotically equals its rank-1 approximation u_iv_i^\top; (iii) these rank-1 matrices are aligned across layers, meaning |v_i+1^\topu_i|→1. In the case of the logistic loss (binary cross entropy), more can be said: the linear function induced by the network --- the product of its weight matrices --- converges to the same direction as the maximum margin solution. This last property was identified in prior work, but only under assumptions on gradient descent which here are implied by the alignment phenomenon.

Citations

• ImageNet classification with deep convolutional neural networks
• Gradient Descent Converges to Minimizers

Cited by

• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
• Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularization
• Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNs
• Multi-step Predictive Coding Leads To Simplicity Bias
• Regularization Implies balancedness in the deep linear network
• Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data
• On the Neural Feature Ansatz for Deep Neural Networks
• PENEX: AdaBoost-Inspired Neural Network Regularization
• From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics
• Low Rank Gradients and Where to Find Them
• Closed-form ℓ_r norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓ_p bias
• Abide by the Law and Follow the Flow: Conservation Laws for Gradient\n Flows

Related

• Directional convergence and alignment in deep learning
• Training invariances and the low-rank phenomenon: beyond linear networks
• On Alignment in Deep Linear Neural Networks
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias