Learning Neural Network Subspaces

2021/02/20 by Mitchell Wortsman, Maxwell Horton, Wortsman, Mitchell +7 · 3 citations
Computer Science · #Machine Learning and Data Classification #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.2102.10472

Abstract

Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.

Citations

• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Underspecification Presents Challenges for Credibility in Modern Machine Learning

Cited by

• Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
• Resting Neurons, Active Insights: Improving Input Sparsification for Large Language Models
• Backdoor Unlearning by Linear Task Decomposition
• SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures

Related

• Training Neural Networks at Any Scale
• Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
• Qualitatively characterizing neural network optimization problems
• Optimization Over Trained Neural Networks: Taking a Relaxing Walk