Deep Probabilistic Ensembles: Approximate Variational Inference through KL Regularization

2018/11/06 by Kashyap Chitta, Chitta, Kashyap, Jose M. Álvarez +3
Computer Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1811.02640

openalex publication_date 2018/11/06 · openalex created_date 2018/11/16 · openalex updated_date 2026/07/28

Abstract

In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep Bayesian Neural Network (BNN). We do so by incorporating a KL divergence penalty term into the training objective of an ensemble, derived from the evidence lower bound used in variational inference. We evaluate the uncertainty estimates obtained from our models for active learning on visual classification. Our approach steadily improves upon active learning baselines as the annotation budget is increased.

Citations

• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Improving neural networks by preventing co-adaptation of feature detectors
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Variational Inference: A Review for Statisticians
• Learning Multiple Layers of Features from Tiny Images

Related

• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Function-Space Regularization for Deep Bayesian Classification
• Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles
• Deep Ensembles from a Bayesian Perspective