Deep Ensembles: A Loss Landscape Perspective

2019/12/05 by Stanislav Fort, Fort, Stanislav, Huiyi Hu +3 · 32 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1912.02757

openalex publication_date 2019/12/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

Citations

• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• Toward Ethical AI Through Bayesian Uncertainty in Neural Question Answering
• ProbeMDE: Uncertainty-Guided Active Proprioception for Monocular Depth Estimation in Surgical Robotics
• Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
• Exploring the Rashomon Set for Concept-Based Models
• Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes
• Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers
• On Joint Regularization and Calibration in Deep Ensembles
• TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
• A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis
• Frequentist Validity of Epistemic Uncertainty Estimators
• Revisiting the Relation Between Robustness and Universality
• Revisiting Hallucination Detection with Effective Rank-based Uncertainty
• High-Rate Mixout: Revisiting Mixout for Robust Domain Generalization
• Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting
• Optical Computing with Spectrally Multiplexed Features in Complex Media
• Deep set based operator learning with uncertainty quantification
• Road Obstacle Video Segmentation
• Hybrid State Estimation of Uncertain Nonlinear Dynamics Using Neural Processes
• Quantifying Out-of-Training Uncertainty of Neural-Network based Turbulence Closures
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversity
• Revisiting Deep Learning Models for Tabular Data
• BLIPs: Bayesian Learned Interatomic Potentials
• TriQDef: Disrupting Semantic and Gradient Alignment to Prevent Adversarial Patch Transferability in Quantized Neural Networks
• Recent advances and applications of deep learning methods in materials science
• SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures
• Diversity and Generalization in Neural Network Ensembles
• Combining Ensembles and Data Augmentation can Harm your Calibration
• ART: Adaptive Relation Tuning for Generalized Relation Prediction
• LVM-GP: Uncertainty-Aware PDE Solver via coupling latent variable model and Gaussian process

Related

• Deep Ensembles Secretly Perform Empirical Bayes
• Deep Ensembles Work, But Are They Necessary?
• On Uniform, Bayesian, and PAC-Bayesian Deep Ensembles
• On Local Posterior Structure in Deep Ensembles
• Deep Ensembles from a Bayesian Perspective