Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

2015/11/19 by Stefan Lee, Senthil Purushwalkam, Lee, Stefan +7 · 29 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Machine Learning and Data Classification #Anomaly Detection Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1511.06314

Abstract

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

Cited by

• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• ProbeMDE: Uncertainty-Guided Active Proprioception for Monocular Depth Estimation in Surgical Robotics
• On Joint Regularization and Calibration in Deep Ensembles
• Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• Objects are Different: Flexible Monocular 3D Object Detection
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Hyperparameter Ensembles for Robustness and Uncertainty Quantification
• Weak-to-Strong Generalization under Distribution Shifts
• BATIS: Bayesian Approaches for Targeted Improvement of Species Distribution Models
• Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting
• Latent Uncertainty Representations for Video-based Driver Action and Intention Recognition
• Quantifying Epistemic Uncertainty in Deep Learning
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Influence of Classification Task and Distribution Shift Type on OOD Detection in Fetal Ultrasound
• Training independent subnetworks for robust prediction
• Deep Ensembles: A Loss Landscape Perspective
• Quantum parameter estimation with uncertainty quantification from continuous measurement data using neural network ensembles
• Uncertainty Estimation using Variance-Gated Distributions
• TRADI: Tracking deep neural network weight distributions for uncertainty\n estimation
• IEA: Inner Ensemble Average within a convolutional neural network
• Deep Extreme Feature Extraction: New MVA Method for Searching Particles in High Energy Physics
• Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Attention-based Ensemble for Deep Metric Learning
• Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance

Related

• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Effective training-time stacking for ensembling of deep neural networks
• Joint Training of Deep Ensembles Fails Due to Learner Collusion
• Snapshot Ensembles: Train 1, get M for free
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity