Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models

2020/12/03 by Xiaofang Wang, Wang, Xiaofang, Dan Kondratyuk +9 · 3 citations
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2012.01988

openalex publication_date 2020/12/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.

Citations

• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
• ImageNet Large Scale Visual Recognition Challenge
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Snapshot Ensembles: Train 1, get M for free
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Short Note about Kinetics-600
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Cited by

• In-Context Distillation with Self-Consistency Cascades: A Simple, Training-Free Way to Reduce LLM Agent Costs
• Semantic Agreement Enables Efficient Open-Ended LLM Cascades
• Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversity

Related

• Committees of deep feedforward networks trained with few data
• Training Efficient CNNS: Tweaking the Nuts and Bolts of Neural Networks for Lighter, Faster and Robust Models
• Differentiable Model Scaling using Differentiable Topk
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning