2022/02/23 by Tim Whitaker, Whitaker, Tim, Darrell Whitley +1 · 3 citations
Computer Science · Mathematics · #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Benchmark (surveying) #Computer science #Domain Adaptation and Few-Shot Learning #Ensemble learning #FOS: Computer and information sciences #Generalization #Machine Learning (cs.LG) #Machine Learning and Data Classification #Machine learning #Mathematics #Pruning #Scratch #cs.LG
paper · pdf · doi:10.48550/arxiv.2202.11782
published in arXiv (Cornell University) (Cornell University) · 12 pages, 7 figures, Accepted to AAAI-22; appendix added
openalex publication_date 2022/02/23 · arxiv created 2022/03/25 · arxiv updated 2022/03/28 · openalex created_date 2022/04/03 · openalex updated_date 2026/08/06
Ensemble Learning is an effective method for improving generalization in machine learning. However, as state-of-the-art neural networks grow larger, the computational cost associated with training several independent networks becomes expensive. We introduce a fast, low-cost method for creating diverse ensembles of neural networks without needing to train multiple models from scratch. We do this by first training a single parent network. We then create child networks by cloning the parent and dramatically pruning the parameters of each child to create an ensemble of members with unique and diverse topologies. We then briefly train each child network for a small number of epochs, which now converge significantly faster when compared to training from scratch. We explore various ways to maximize diversity in the child networks, including the use of anti-random pruning and one-cycle tuning. This diversity enables "Prune and Tune" ensembles to achieve results that are competitive with traditional ensembles at a fraction of the training cost. We benchmark our approach against state of the art low-cost ensemble methods and display marked improvement in both accuracy and uncertainty estimation on CIFAR-10 and CIFAR-100.