Bayesian Neural Network Priors Revisited

2021/02/12 by Vincent Fortuin, Adrià Garriga-Alonso, Fortuin, Vincent +13 · 24 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Bayesian inference #Bayesian probability #Computer science #Convolutional neural network #FOS: Computer and information sciences #Gaussian #Gaussian Processes and Bayesian Inference #Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine learning #Pattern recognition (psychology) #Prior probability #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2102.06571

published in arXiv (Cornell University) (Cornell University) · Accepted at ICLR 2022

openalex publication_date 2021/02/12 · arxiv created 2022/03/16 · arxiv updated 2022/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Isotropic Gaussian priors are the de facto standard for modern Bayesian neural network inference. However, it is unclear whether these priors accurately reflect our true beliefs about the weight distributions or give optimal performance. To find better priors, we study summary statistics of neural network weights in networks trained using stochastic gradient descent (SGD). We find that convolutional neural network (CNN) and ResNet weights display strong spatial correlations, while fully connected networks (FCNNs) display heavy-tailed weight distributions. We show that building these observations into priors can lead to improved performance on a variety of image classification datasets. Surprisingly, these priors mitigate the cold posterior effect in FCNNs, but slightly increase the cold posterior effect in ResNets.

Citations

• A Scale Mixture Perspective of Multiplicative Noise in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Vergleichung von zwei Werthen des wahrscheinlichen Fehlers
• Strictly Proper Scoring Rules, Prediction, and Estimation
• Sparse coding with an overcomplete basis set: A strategy employed by V1?
• A Practical Bayesian Framework for Backpropagation Networks
• Gradient-based learning applied to document recognition
• Variational Dropout Sparsifies Deep Neural Networks
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Jug: Software for Parallel Reproducible Computation in Python
• Functional Variational Bayesian Neural Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Weight Uncertainty in Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Richer priors for infinitely wide multi-layer perceptrons
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Learning Multiple Layers of Features from Tiny Images
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

Cited by

• Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance
• From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network
• Tempering the Bayes Filter towards Improved Model-Based Estimation
• Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks
• Partial Trace-Class Bayesian Neural Networks
• The Sensitivity of Variational Bayesian Neural Network Performance to Hyperparameters
• Amortising Inference and Meta-Learning Priors in Neural Networks
• Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning
• Simulating Posterior Bayesian Neural Networks with Dependent Weights
• Textual Bayes: Quantifying Uncertainty in LLM-Based Systems
• BackSlash: Rate Constrained Optimized Training of Large Language Models

Related

• Structured Weight Priors for Convolutional Neural Networks
• A statistical theory of cold posteriors in deep neural networks
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• A Bayesian Perspective on the Deep Image Prior
• The Deep Weight Prior