Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

2015/06/06 by Yarin Gal, Zoubin Ghahramani, Gal, Yarin +1 · 27 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1506.02158

openalex publication_date 2015/06/06 · openalex created_date 2016/06/24 · openalex updated_date 2026/07/28

Abstract

Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to over-fitting on small data than traditional approaches. This is by placing a probability distribution over the CNN's kernels. We approximate our model's intractable posterior with Bernoulli variational distributions, requiring no additional model parameters. On the theoretical side, we cast dropout network training as approximate inference in Bayesian neural networks. This allows us to implement our model using existing tools in deep learning with no increase in time complexity, while highlighting a negative result in the field. We show a considerable improvement in classification accuracy compared to standard techniques and improve on published state-of-the-art results for CIFAR-10.

Citations

• A Practical Bayesian Framework for Backpropagation Networks
• Gradient-based learning applied to document recognition
• Variational Bayesian Inference with Stochastic Search
• ImageNet classification with deep convolutional neural networks
• Deeply-Supervised Nets
• Weight Uncertainty in Neural Networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Bayes-DIC Net: Estimating Digital Image Correlation Uncertainty with Bayesian Neural Networks
• Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation
• Natural gradient descent for improving variational inference based classification of radio galaxies
• Optimizing Uncertainty-Aware Deep Learning for On-the-Edge Murmur Detection in Low-Resource Settings
• Bayesian Topological Convolutional Neural Nets
• Test-time Uncertainty Estimation for Medical Image Registration via Transformation Equivariance
• Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition
• Neovascularization Segmentation via a Multilateral Interaction-Enhanced Graph Convolutional Network
• QUTCC: Quantile Uncertainty Training and Conformal Calibration for Imaging Inverse Problems
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization
• A decision-theoretic approach for model interpretability in Bayesian framework
• Uncertainty-Aware Label Refinement for Sequence Labeling
• Stochastic-Shield: A Probabilistic Approach Towards Training-Free Adversarial Defense in Quantized CNNs
• VidLoc: A Deep Spatio-Temporal Model for 6-DoF Video-Clip Relocalization
• Deep Probabilistic Ensembles: Approximate Variational Inference through KL Regularization
• Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
• Semi-Supervised Domain Generalization with Stochastic StyleMatch
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Towards real-time assessment of infrasound event detection capability using deep learning-based transmission loss estimation
• Active Learning Solution on Distributed Edge Computing
• Towards Robust Evaluations of Continual Learning
• Deep Learning and Bayesian Deep Learning Based Gender Prediction in Multi-Scale Brain Functional Connectivity
• Image segmentation of liver stage malaria infection with spatial uncertainty sampling

Related

• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Robustly representing uncertainty in deep neural networks through sampling
• Variational Bayesian Bow tie Neural Networks with Shrinkage
• Variational Bayesian dropout: pitfalls and fixes
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization