Essentially No Barriers in Neural Network Energy Landscape

2018/03/31 by Felix Draxler, Kambis Veschgini, Manfred Salmhofer +1 · 2 citations
Mathematics · Computer Science · #stat.ML #cs.AI #cs.LG

paper · pdf

published as Proceedings of the 35th International Conference on Machine Learning, PMLR 80:1308-1317, 2018 · In Proceedings of 35th International Conference on Machine Learning (ICML 2018)

arxiv created 2019/02/22 · arxiv updated 2019/02/25

Abstract

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

Cited by

• A Combinatorial Theory of Dropout: Subnetworks, Graph Geometry, and Generalization
• Convergent Evolution in Algorithmic Space
• Single-Input Multi-Output Model Merging: Leveraging Foundation Models for Dense Multi-Task Learning
• LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks