Convergence beyond the over-parameterized regime using Rayleigh quotients

2023/01/19 by D. Robin, Robin, David A. R., Kevin Scaman +3 · 1 citation
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Algorithms #Model Reduction and Neural Networks #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2301.08117

openalex publication_date 2023/01/19 · openalex created_date 2023/01/21 · openalex updated_date 2026/08/01

Abstract

In this paper, we present a new strategy to prove the convergence of deep learning architectures to a zero training (or even testing) loss by gradient flow. Our analysis is centered on the notion of Rayleigh quotients in order to prove Kurdyka-Łojasiewicz inequalities for a broader set of neural network architectures and loss functions. We show that Rayleigh quotients provide a unified view for several convergence analysis techniques in the literature. Our strategy produces a proof of convergence for various examples of parametric learning. In particular, our analysis does not require the number of parameters to tend to infinity, nor the number of samples to be finite, thus extending to test loss minimization and beyond the over-parameterized regime.

Cited by

• Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case

Related

• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Gradient Flow Convergence Guarantee for General Neural Network Architectures
• A Framework for Overparameterized Learning
• Loss Landscape Characterization of Neural Networks without Over-Parametrization
• On the Convergence of Gradient Descent for Large Learning Rates