Training independent subnetworks for robust prediction

2020/10/13 by Marton Havasi, Rodolphe Jenatton, Havasi, Marton +13 · 2 voices · 9 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2010.06610

openalex publication_date 2020/10/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant computational cost. In this work, we show a surprising result: the benefits of using multiple predictions can be achieved `for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100, ImageNet, and their out-of-distribution variants compared to previous methods.

Citations

• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

Cited by

• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• TGSFormer: Scalable Temporal Gaussian Splatting for Embodied Semantic Scene Completion
• Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion
• Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification
• On Joint Regularization and Calibration in Deep Ensembles
• Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
• Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
• Understanding Uncertainty in Bayesian Deep Learning
• Online Ensemble Transformer for Accurate Cloud Workload Forecasting in Predictive Auto-Scaling

Discussions

• Training independent subnetworks for robust prediction [hn, 9 points, 0 comments]
• Very interesting! You may be interested in this related approach: "Training independent subnetworks for robust prediction" arxiv.org/abs/2010.06610 [bsky, 0 points, 1 comments]

Related

• Robustness via Cross-Domain Ensembles
• Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning
• Snapshot Ensembles: Train 1, get M for free
• Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks
• Sequential Bayesian Neural Subnetwork Ensembles