Sparse MoEs meet Efficient Ensembles

2021/10/07 by James Urquhart Allingham, Allingham, James Urquhart, Florian Wenzel +23
Computer Science · Engineering · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.2110.03360

openalex publication_date 2021/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present Efficient Ensemble of Experts (E³), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E³ over several challenging vision Transformer-based baselines. E³ not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.

Citations

• Distilling the Knowledge in a Neural Network
• Hierarchical Mixtures of Experts and the EM Algorithm
• Neural Networks and the Bias/Variance Dilemma
• Adaptive Mixtures of Local Experts
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Natural Adversarial Examples
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Language Models are Few-Shot Learners
• Deep Ensembles for Low-Data Transfer Learning
• Learning Multiple Layers of Features from Tiny Images
• Learning Transferable Visual Models From Natural Language Supervision
• Revisiting the Calibration of Modern Neural Networks
• Exploring the Limits of Out-of-Distribution Detection

Related

• From Sparse to Soft Mixtures of Experts
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
• Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
• Faster MoE LLM Inference for Extremely Large Models
• Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts