Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

2015/11/19 by Stefan Lee, Senthil Purushwalkam, Lee, Stefan +7 · 1 voice · 204 citations
Computer Science · Engineering · #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Class (philosophy) #Computer science #Convolutional neural network #Deep neural networks #Domain Adaptation and Few-Shot Learning #Engineering #Ensemble forecasting #Ensemble learning #Initialization #Machine Learning and Data Classification #Machine learning #Oracle #Range (aeronautics) #Variation (astronomy) #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1511.06314

published in arXiv (Cornell University) (Cornell University)

arxiv created 2015/11/19 · openalex publication_date 2015/11/19 · arxiv published 2015/11/19 · arxiv updated 2015/11/20 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

Cited by

• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• ProbeMDE: Uncertainty-Guided Active Proprioception for Monocular Depth Estimation in Surgical Robotics
• On Joint Regularization and Calibration in Deep Ensembles
• Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• Objects are Different: Flexible Monocular 3D Object Detection
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Weak-to-Strong Generalization under Distribution Shifts
• BATIS: Bayesian Approaches for Targeted Improvement of Species Distribution Models
• Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting
• Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition
• Quantifying Epistemic Uncertainty in Deep Learning
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Influence of Classification Task and Distribution Shift Type on OOD Detection in Fetal Ultrasound
• Training independent subnetworks for robust prediction
• Deep Ensembles: A Loss Landscape Perspective
• Quantum parameter estimation with uncertainty quantification from continuous measurement data using neural network ensembles
• Uncertainty Estimation using Variance-Gated Distributions
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• IEA: Inner Ensemble Average within a convolutional neural network
• Deep Extreme Feature Extraction: New MVA Method for Searching Particles in High Energy Physics
• Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Attention-based Ensemble for Deep Metric Learning
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
• Speed/accuracy trade-offs for modern convolutional object detectors
• Regularizing Neural Networks via Stochastic Branch Layers
• Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet
• Last Layer Hamiltonian Monte Carlo
• Deep Ensembles for Low-Data Transfer Learning
• WWAggr: A Window Wasserstein-based Aggregation for Ensemble Change Point Detection
• Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System
• Sparse MoEs meet Efficient Ensembles
• Probabilistic Regression of Rotations using Quaternion Averaging and a Deep Multi-Headed Network
• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• Asymmetric Duos: Sidekicks Improve Uncertainty
• NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Deep Architectures and Ensembles for Semantic Video Classification
• Quality Resilient Deep Neural Networks
• Demystifying Parallel and Distributed Deep Learning
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
• Robust Semantic Segmentation with Superpixel-Mix
• Last-layer committee machines for uncertainty estimations of benthic imagery
• Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?

Discussions

• Why M Heads Are Better Than One: Training a Diverse Ensemble of Deep Networks [hn, 2 points, 0 comments]

Related

• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Effective training-time stacking for ensembling of deep neural networks
• Joint Training of Deep Ensembles Fails Due to Learner Collusion
• Snapshot Ensembles: Train 1, get M for free
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity