Priors in Bayesian Deep Learning: A Review

2021/05/14 by Vincent Fortuin, Fortuin, Vincent · 20 citations
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2105.06868

openalex publication_date 2021/05/14 · arxiv created 2022/03/18 · arxiv updated 2022/03/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.

Cited by

• Lifting Biomolecular Data Acquisition
• Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning
• Online Learning of Modular Bayesian Deep Receivers: Single-Step Adaptation with Streaming Data
• Partial Trace-Class Bayesian Neural Networks
• To MCMC or not to MCMC: Evaluating non-MCMC methods for Bayesian penalized regression
• Guided Uncertainty Learning Using a Post-Hoc Evidential Meta-Model
• Incorporating priors in learning: a random matrix study under a teacher-student framework
• Amortising Inference and Meta-Learning Priors in Neural Networks
• Anchored Langevin Algorithms
• Simulation Priors for Data-Efficient Deep Learning
• Hi-fi functional priors by learning activations
• Diff-ANO: Towards Fast High-Resolution Ultrasound Computed Tomography via Conditional Consistency Models and Adjoint Neural Operators
• Adaptive Parameter Optimization in Gaussian Processes: A Comprehensive Study of Uncertainty Quantification and Dimensional Scaling
• laplax -- Laplace Approximations with JAX
• Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'
• Robust Adversarial Quantification via Conflict-Aware Evidential Deep Learning
• Position: Epistemic uncertainty estimation methods are fundamentally incomplete
• Variational Deep Learning via Implicit Regularization
• Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review
• Normalizing Flow Regression for Bayesian Inference with Offline Likelihood Evaluations

Related

• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Deep Prior
• Deep Reference Priors: What is the best way to pretrain a model?
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• The prior can generally only be understood in the context of the likelihood