Relative Flatness and Generalization

2020/01/03 by Henning Petzka, Petzka, Henning, Michael Kamp +7 · 19 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Algorithm #Applied mathematics #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Flatness (cosmology) #Generalization #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical analysis #Mathematics #Physics #Robustness (evolution) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2001.00939

published in arXiv (Cornell University) 34 (Cornell University) · The first two authors made equal contribution; Accepted for publication at NeurIPS 2021; arXiv admin note: substantial text overlap with arXiv:1912.00058

openalex publication_date 2020/01/03 · arxiv created 2021/11/04 · arxiv updated 2021/11/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating it to the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and to identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flatness measure that correlates strongly with generalization, simplifies to ridge regression for ordinary least squares, and solves the reparameterization issue.

Citations

• Three Factors Influencing Minima in SGD
• Identifying Generalization Properties in Neural Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Learning Multiple Layers of Features from Tiny Images
• A Reparameterization-Invariant Flatness Measure for Deep Neural Networks

Cited by

• Are Flat Minima an Illusion?
• Model Merging via Multi-Teacher Knowledge Distillation
• Task Specific Sharpness Aware O-RAN Resource Management using Multi Agent Reinforcement Learning
• Zero-Order Sharpness-Aware Minimization
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• Generalization Below the Edge of Stability: The Role of Data Geometry
• When Flatness Does (Not) Guarantee Adversarial Robustness
• A Function-Centric Perspective on Flat and Sharp Minima
• Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
• Dynamics of Learning: Generative Schedules from Latent ODEs
• Simplifying Neural Networks During Training
• Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking
• Curvature Learning for Generalization of Hyperbolic Neural Networks
• Simple Optimizers for Convex Aligned Multi-Objective Optimization
• Flatness After All?
• Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models
• Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It
• Connecting Independently Trained Modes via Layer-Wise Connectivity
• FIC-TSC: Learning Time Series Classification with Fisher Information Constraint

Related

• Flatness After All?
• FAM: Relative Flatness Aware Minimization
• A Reparameterization-Invariant Flatness Measure for Deep Neural Networks
• Why flatness does and does not correlate with generalization for deep neural networks
• Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization