Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

2019/06/14 by Rohith Kuditipudi, Xiang Wang, Kuditipudi, Rohith +13 · 13 citations
Computer Science · Environmental Science · Mathematics · #Computer science #Data Visualization and Analytics #FOS: Computer and information sciences #Geography #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Remote Sensing and LiDAR Applications #Wildlife-Road Interactions and Conservation #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.06247

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2019/06/14 · arxiv created 2020/01/06 · arxiv updated 2020/01/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segments. We give mathematical explanations for this phenomenon, assuming generic properties (such as dropout stability and noise stability) of well-trained deep nets, which have previously been identified as part of understanding the generalization properties of deep nets. Our explanation holds for realistic multilayer nets, and experiments are presented to verify the theory.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Loss Surfaces of Multilayer Networks
• On the importance of single directions for generalization
• On the loss landscape of a class of deep neural networks with no bad local valleys

Cited by

• A Systematic Study of In-the-Wild Model Merging for Large Language Models
• Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity
• SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging
• Understanding Mode Connectivity via Parameter Space Symmetry
• On Learning Verifiers and Implications to Chain-of-Thought Reasoning
• Connecting Independently Trained Modes via Layer-Wise Connectivity

Related

• When Are Solutions Connected in Deep Networks?
• Landscaping Linear Mode Connectivity
• Input Space Mode Connectivity in Deep Neural Networks
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Large Scale Structure of Neural Network Loss Landscapes