Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

2021/05/25 by Şimşek, Berfin, Ged, François, Jacot, Arthur +4 · 9 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2105.12221

Abstract

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with L layers of minimal widths r₁^*, …, r_L-1^* reaches a zero-loss minimum at r₁^*! ⋯ r_L-1^*! isolated points that are permutations of one another, we show that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold. For a two-layer overparameterized network of width r^*+ h =: m we explicitly describe the manifold of global minima: it consists of T(r^*, m) affine subspaces of dimension at least h that are connected to one another. For a network of width m, we identify the number G(r,m) of affine subspaces containing only symmetry-induced critical points that are related to the critical points of a smaller network of width r

Cited by

• Teleportation-Based Defenses for Privacy in Approximate Machine Unlearning
• Fundamentals of Regression
• Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queries
• A mathematical theory for understanding when abstract representations emerge in neural networks
• Architecture Induces Structural Invariant Manifolds of Neural Network Training Dynamics
• Do We Really Need Permutations? Impact of Width Expansion on Linear Mode Connectivity
• Riddled basin geometry sets fundamental limits to predictability and reproducibility in deep learning
• Self-organized learning emerges from coherent coupling of critical neurons
• Saddle Hierarchy in Dense Associative Memory

Related

• Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• The loss landscape of overparameterized neural networks
• The critical locus of overparameterized neural networks
• Understanding Mode Connectivity via Parameter Space Symmetry