On the interplay between noise and curvature and its effect on optimization and generalization

2019/06/18 by Valentin Thomas, Thomas, Valentin, Fabián Pedregosa +10 · 11 citations
Computer Science · Engineering · Mathematics · #Face and Expression Recognition #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.07774

Accepted to AISTATS 2020

arxiv created 2020/04/06 · arxiv updated 2020/04/08

Abstract

The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ultimate goal is good generalization performance, we clarify how both curvature and noise are relevant to properly estimate the generalization gap. Realizing that the limitations of some existing works stems from a confusion between these matrices, we also clarify the distinction between the Fisher matrix, the Hessian, and the covariance matrix of the gradients.

Citations

• Multimodel Inference
• Three Factors Influencing Minima in SGD
• Real numbers, data science and chaos: How to fit any dataset with a single parameter

Cited by

• Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
• TPV: Parameter Perturbations Through the Lens of Test Prediction Variance
• A Bootstrap Perspective on Stochastic Gradient Descent
• A Generalization Gap Estimation for Overparameterized Models via the Langevin Functional Variance
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics
• Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator
• On the Noisy Gradient Descent that Generalizes as SGD
• L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
• Flatness After All?
• Optimization and variability can coexist
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning

Related

• Escaping Saddles with Stochastic Gradients
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent
• Second-Order Guarantees of Stochastic Gradient Descent in Non-Convex\n Optimization
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of\n Escaping from Sharp Minima and Regularization Effects