Solving hybrid machine learning tasks by traversing weight space\n geodesics

2021/06/05 by Guruprasad Raghavan, Matt Thomson, Raghavan, Guruprasad +1
Computer Science · Engineering · #3D Shape Modeling and Analysis #Advanced Numerical Analysis Techniques #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Medical Imaging and Analysis

paper · pdf · doi:10.48550/arxiv.2106.02793

openalex publication_date 2021/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning problems have an intrinsic geometric structure as central\nobjects including a neural network's weight space and the loss function\nassociated with a particular task can be viewed as encoding the intrinsic\ngeometry of a given machine learning problem. Therefore, geometric concepts can\nbe applied to analyze and understand theoretical properties of machine learning\nstrategies as well as to develop new algorithms. In this paper, we address\nthree seemingly unrelated open questions in machine learning by viewing them\nthrough a unified framework grounded in differential geometry. Specifically, we\nview the weight space of a neural network as a manifold endowed with a\nRiemannian metric that encodes performance on specific tasks. By defining a\nmetric, we can construct geodesic, minimum length, paths in weight space that\nrepresent sets of networks of equivalent or near equivalent functional\nperformance on a specific task. We, then, traverse geodesic paths while\nidentifying networks that satisfy a second objective. Inspired by the geometric\ninsight, we apply our geodesic framework to 3 major applications: (i) Network\nsparsification (ii) Mitigating catastrophic forgetting by constructing networks\nwith high performance on a series of objectives and (iii) Finding high-accuracy\npaths connecting distinct local optima of deep networks in the non-convex loss\nlandscape. Our results are obtained on a wide range of network architectures\n(MLP, VGG11/16) trained on MNIST, CIFAR-10/100. Broadly, we introduce a\ngeometric framework that unifies a range of machine learning objectives and\nthat can be applied to multiple classes of neural network architectures.\n

Citations

• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Qualitatively characterizing neural network optimization problems
• The Loss Surfaces of Multilayer Networks
• Natural Gradient Works Efficiently in Learning
• Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• On Geometric Structure of Activation Spaces in Neural Networks
• On the distance between two neural networks and the stability of learning

Related

• Sparsifying networks by traversing Geodesics
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure
• A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold
• Geodesic convolutional neural networks on Riemannian manifolds