Weight Uncertainty in Neural Networks

2015/05/20 by Charles Blundell, Blundell, Charles, Julien Cornebise +5 · 1,284 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Backpropagation #Bayes' theorem #Bayesian probability #Computer science #Dropout (neural networks) #Energy (signal processing) #FOS: Computer and information sciences #MNIST database #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematical optimization #Mathematics #Reinforcement Learning in Robotics #Reinforcement learning #Statistics #Upper and lower bounds #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1505.05424

published in arXiv (Cornell University) (Cornell University) · In Proceedings of the 32nd International Conference on Machine Learning (ICML 2015)

openalex publication_date 2015/05/20 · arxiv created 2015/05/21 · arxiv updated 2015/05/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Auto-Encoding Variational Bayes
• A Practical Bayesian Framework for Backpropagation Networks

Cited by

• DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertainty
• Transfer Learning for Analysis of Collective and Non-Collective Thomson Scattering Spectra
• Toward Ethical AI Through Bayesian Uncertainty in Neural Question Answering
• Unreliable Uncertainty Estimates with Monte Carlo Dropout
• Selective Conformal Risk Control
• PIS: A Generalized Physical Inversion Solver for Arbitrary Sparse Observations via Set Conditioned Flow Matching
• Uncertainty-Preserving QBNNs: Multi-Level Quantization of SVI-Based Bayesian Neural Networks for Image Classification
• Minimization of Functions on Dually Flat Spaces Using Geodesic Descent Based on Dual Connections
• Bayesian Physics-Informed Neural Networks for Inverse Problems (BPINN-IP): Application in Infrared Image Processing
• Uncertainty Reasoning with Photonic Bayesian Machines
• Probabilistic Neuro-Symbolic Reasoning for Sparse Historical Data: A Framework Integrating Bayesian Inference, Causal Models, and Game-Theoretic Allocation
• Nanosecond-Scale Proton Emission from Triaxially Deformed Lu-148 Predicted with High Accuracy Qp Value via Novel Bayesian Evaluation
• Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation
• PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic
• Training-Free Active Learning Framework in Materials Science with Large Language Models
• Natural gradient descent for improving variational inference based classification of radio galaxies
• Bayesian--AI Fusion for Epidemiological Decision Making: Calibrated Risk, Honest Uncertainty, and Hyperparameter Intelligence
• Uncertainty Calibration of Multi-Label Bird Sound Classifiers
• Uncertainty Guided Online Ensemble for Non-stationary Data Streams in Fusion Science
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• Uncertainty-Guided LLM Semantic Augmentation for Heterogeneous Treatment Effect Estimation
• The Bayesian Geometry of Transformer Attention
• FruitProm: Probabilistic Maturity Estimation and Detection of Fruits and Vegetables
• VIKING: Deep variational inference with stochastic projections
• Interpretable epistemic uncertainty decomposition in sequential generative models via polynomial chaos surrogates
• Functional Distribution Networks (FDN)
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• Bayesian Topological Convolutional Neural Nets
• A Systematic Literature Review of Machine Learning Techniques for Observational Constraints in Cosmology
• Robust Classification of Oral Cancer with Limited Training Data
• Bayesian Surrogates for Risk-Aware Pre-Assessment of Aging Bridge Portfolios
• Uncertainty-Aware Deep Learning for Wildfire Danger Forecasting
• Tracing Uncertainty in Language Model "Reasoning"
• Learnable Conformal Prediction with Context-Aware Nonconformity Functions for Robotic Planning and Perception
• Multi-modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation
• Approaching Neural Network Uncertainty Realism
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Uncertainty Estimation in Deep Neural Networks for Point Cloud Segmentation in Factory Planning
• Confidence-Aware Routing for Large Language Model Reliability Enhancement: A Multi-Signal Approach to Pre-Generation Hallucination Mitigation
• Towards Open-Ended Discovery for Low-Resource NLP
• A Unified Particle-Optimization Framework for Scalable Bayesian Sampling
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
• Information Geometry of Variational Bayes
• Being Bayesian about Categorical Probability
• A Conformal Prediction Framework for Uncertainty Quantification in Physics-Informed Neural Networks
• RCoNet: Deformable Mutual Information Maximization and High-order Uncertainty-aware Learning for Robust COVID-19 Detection
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Unifying Variational Inference and PAC-Bayes for Supervised Learning that Scales
• MC²RAM: Markov Chain Monte Carlo Sampling in SRAM for Fast Bayesian Inference
• Action and Perception as Divergence Minimization
• Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Sparsifying Neural Network Connections for Face Recognition
• Peer-to-peer Federated Learning on Graphs
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• A probabilistic approach to tomography and adjoint state methods, with an application to full waveform inversion in medical ultrasound
• Data Uncertainty Learning in Face Recognition
• Uncertainty based Class Activation Maps for Visual Question Answering
• Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentation
• Compressing Neural Networks using the Variational Information Bottleneck
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Masked Contrastive Learning for Anomaly Detection
• Diversity inducing Information Bottleneck in Model Ensembles
• Sampling-based Bayesian Inference with gradient uncertainty
• Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
• Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Improving Local Identifiability in Probabilistic Box Embeddings
• HierCVAE: Hierarchical Attention-Driven Conditional Variational Autoencoders for Multi-Scale Temporal Modeling
• Approximate Bayesian Optimisation for Neural Networks
• Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
• NP-ODE: Neural Process Aided Ordinary Differential Equations for Uncertainty Quantification of Finite Element Analysis
• Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Uncertainty Tube Visualization of Particle Trajectories
• BLIPs: Bayesian Learned Interatomic Potentials
• Understanding Uncertainty in Bayesian Deep Learning
• Verifying the Causes of Adversarial Examples
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
• Deep density networks and uncertainty in recommender systems
• Improving Back-Translation with Uncertainty-based Confidence Estimation
• Deep kernel processes
• Routing Networks with Co-training for Continual Learning
• On Hallucination and Predictive Uncertainty in Conditional Language Generation
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• Bayesian Distributional Policy Gradients
• Stochastic Maximum Likelihood Optimization via Hypernetworks
• Explainable Image Classification with Reduced Overconfidence for Tissue Characterisation
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Bayesian Graph Neural Networks for Molecular Property Prediction
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization
• Benchmarking the Neural Linear Model for Regression
• B-SCST: Bayesian Self-Critical Sequence Training for Image Captioning
• Probabilistic neural network-based reduced-order surrogate for fluid flows
• Interval Neural Networks: Uncertainty Scores
• Transferable Calibration with Lower Bias and Variance in Domain Adaptation
• Adversarial Distillation of Bayesian Neural Network Posteriors
• On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
• Adaptively Preconditioned Stochastic Gradient Langevin Dynamics
• Discrete flow posteriors for variational inference in discrete dynamical systems
• Safety and Robustness in Decision Making: Deep Bayesian Recurrent Neural Networks for Somatic Variant Calling in Cancer
• Towards calibrated and scalable uncertainty representations for neural networks
• Neural Thompson Sampling
• Old Rules in a New Game: Mapping Uncertainty Quantification to Quantum Machine Learning
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Uncertainty Quantification for Machine Learning-Based Prediction: A Polynomial Chaos Expansion Approach for Joint Model and Input Uncertainty Propagation
• Interpretable Sequence Learning for COVID-19 Forecasting
• Comparison of Graphcore IPUs and Nvidia GPUsfor cosmology applications
• Stochastic Hyperparameter Optimization through Hypernetworks
• Efficient and Robust Machine Learning for Real-World Systems
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• Amortized Variational Deep Q Network
• Variational Information Bottleneck Model for Accurate Indoor Position Recognition
• Model Free Barrier Functions via Implicit Evading Maneuvers
• Bayesian Deep Learning for Convective Initiation Nowcasting Uncertainty Estimation
• Variational Implicit Processes
• Pixel Invisibility: Detecting Objects Invisible in Color Images
• Improving predictions of Bayesian neural nets via local linearization
• A Physics-Informed Data-Driven Discovery for Constitutive Modeling of Compressible, Nonlinear, History-Dependent Soft Materials under Multiaxial Cyclic Loading
• Low Complexity Approximate Bayesian Logistic Regression for Sparse Online Learning
• Differentially Private Conformal Prediction via Quantile Binary Search
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Probabilistic Discriminative Learning with Layered Graphical Models
• Semi-Supervised Hypothesis Transfer for Source-Free Domain Adaptation
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Robustness of Bayesian Neural Networks to Gradient-Based Attacks
• Adaptive Bayesian Reticulum
• Probabilistic prediction of the heave motions of a semi-submersible by a deep learning problem model
• Neural Likelihoods for Multi-Output Gaussian Processes
• Task Agnostic Continual Learning Using Online Variational Bayes
• Generating Neural Networks with Neural Networks
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning
• Gradient-Free Adversarial Attacks for Bayesian Neural Networks
• Bayesian active learning for production, a systematic study and a reusable library
• Deep Bayesian Active Learning for Multiple Correct Outputs
• The Bayesian Approach to Continual Learning: An Overview
• Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
• CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk
• Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation
• Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization
• Notes on the Behavior of MC Dropout
• Hypermodels for Exploration
• Variance Networks: When Expectation Does Not Meet Your Expectations
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Variational Graph Convolutional Neural Networks
• Continual Learning Using Bayesian Neural Networks
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Multi-Task Variational Information Bottleneck
• Bayesian Neural Networks with Soft Evidence
• Efficient Action Recognition Using Confidence Distillation
• Multi-Model Framework for Reconstructing Gamma-Ray Burst Light Curves
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Learning Adapter Rank via Symmetry Breaking
• Sampling-Free Learning of Bayesian Quantized Neural Networks
• TITAN: Query-Token based Domain Adaptive Adversarial Learning
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Constrained Structured Regression with Convolutional Neural Networks
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Cross-regularization: Adaptive Model Complexity through Validation Gradients
• Bayesian x-vector: Bayesian Neural Network based x-vector System for Speaker Verification
• Uncertainty-Aware Lookahead Factor Models for Quantitative Investing
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Attended Temperature Scaling: A Practical Approach for Calibrating Deep Neural Networks
• T-SVDNet: Exploring High-Order Prototypical Correlations for Multi-Source Domain Adaptation
• FeTa: A DCA Pruning Algorithm with Generalization Error Guarantees
• Uncertainty Estimation by Human Perception versus Neural Models
• Combining Model and Parameter Uncertainty in Bayesian Neural Networks
• JAXNS: a high-performance nested sampling package based on JAX
• Variational Learning Finds Flatter Solutions at the Edge of Stability
• Sequential Anomaly Detection using Inverse Reinforcement Learning
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Distribution-Level AirComp for Wireless Federated Learning under Data Scarcity and Heterogeneity
• Uncertainty quantification of molecular property prediction using Bayesian neural network models
• Uncertainty-guided active learning for surrogate prediction of stream-finishing wear fields
• Probabilistic Model-Agnostic Meta-Learning
• Bayesian Compression for Deep Learning
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
• Generative Particle Variational Inference via Estimation of Functional Gradients
• BAR: Bayesian Activity Recognition using variational inference
• Bayesian deep learning of affordances from RGB images
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Radial and Directional Posteriors for Bayesian Neural Networks
• Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics
• Efficient exploration with Double Uncertain Value Networks
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• On the Need to Align Intent and Implementation in Uncertainty Quantification for Machine Learning
• Variational Bayesian Quantization
• Towards Robust Evaluations of Continual Learning
• Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
• Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles
• Bayesian Autoencoders: Analysing and Fixing the Bernoulli likelihood for Out-of-Distribution Detection
• Inherent Noise in Gradient Based Methods
• Physics-Constrained Bayesian Neural Network for Fluid Flow Reconstruction with Sparse and Noisy Data
• Fast Adaptive Weight Noise
• Uncertainty Estimation for Heterophilic Graphs Through the Lens of Information Theory
• CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment
• Structured Bayesian Gaussian process latent variable model
• Variational Deep Learning via Implicit Regularization
• Class Incremental Online Streaming Learning
• Cooperative Variance Estimation and Bayesian Neural Networks for Disentangling Aleatoric and Epistemic Uncertainties
• On Machine Learning and Structure for Mobile Robots
• VFunc: a Deep Generative Model for Functions
• Physics-Informed Neural Networks for Sparse Strain-Field Reconstruction in 4D-STEM
• A Bayesian Approach to Invariant Deep Neural Networks
• Predictive Uncertainty Quantification with Compound Density Networks
• Decision Theoretic Bootstrapping
• Logit-based Uncertainty Measure in Classification
• Stochastic Weight Sharing for Bayesian Neural Networks
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• Constrained Co-Design for Photonic Bayesian Neural Networks
• Uncertainty quantification of molecular property prediction with Bayesian neural networks
• Fast learning rate of deep learning via a kernel perspective
• Bayesian Variational Optimization for Combinatorial Spaces
• Sampling Prediction-Matching Examples in Neural Networks: A Probabilistic Programming Approach
• Estimation with Uncertainty via Conditional Generative Adversarial Networks
• Meta Learning Black-Box Population-Based Optimizers
• Stochastic Gradient MCMC with Stale Gradients
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• Uncertainty-Aware Multiple Instance Learning from Large-Scale Long Time Series Data
• Bayesian Deep Learning Approaches for Uncertainty-Aware Retinal OCT Image Segmentation for Multiple Sclerosis
• Trust Region Value Optimization using Kalman Filtering
• A Physics-Data-Driven Bayesian Method for Heat Conduction Problems
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• Recursive Gaussian Processes and the Bayesian Brain
• Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency
• A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks
• Eigenvalue Corrected Noisy Natural Gradient
• Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
• Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Probabilistic Wind Power Modelling via Heteroscedastic Non-Stationary Gaussian Processes
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
• Learning Neural Networks with Adaptive Regularization
• Improved Uncertainty Quantification in Physics-Informed Neural Networks Using Error Bounds and Solution Bundles
• Modeling continuous-time stochastic processes using \N-Curve\n mixtures
• Bridging the Gap Between Explainable AI and Uncertainty Quantification to Enhance Trustability
• Bayesian Recurrent Framework for Missing Data Imputation and Prediction with Clinical Time Series
• Uncertainty Quantification for Machine Learning in Healthcare: A Survey
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
• Safe end-to-end imitation learning for model predictive control
• Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
• A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
• Variational Resampling Based Assessment of Deep Neural Networks under Distribution Shift
• Learning the Standard Model Manifold: Bayesian Latent Diffusion for Collider Anomaly Detection
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Scalable Uncertainty Quantification for Extreme Weather Forecasting via Empirical Neural Tangent Kernels
• On the generalization of bayesian deep nets for multi-class classification
• Deep Bayesian Recurrent Neural Networks for Somatic Variant Calling in Cancer
• Last-layer committee machines for uncertainty estimations of benthic imagery
• Bayesian neural networks at scale: a performance analysis and pruning study
• On Batch Normalisation for Approximate Bayesian Inference
• Context Aware Grounded Teacher for Source Free Object Detection
• Tied Hidden Factors in Neural Networks for End-to-End Speaker Recognition
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• Approximate Bayesian Uncertainties on Deep Learning Dynamical Mass Estimates of Galaxy Clusters
• Bayesian continual learning and forgetting in neural networks
• Bayesian Neural Network Priors Revisited
• Variational Bayes: A report on approaches and applications
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Simultaneously Reconciled Quantile Forecasting of Hierarchically Related Time Series
• An Equivalence of Fully Connected Layer and Convolutional Layer
• Probabilistic Machine Learning for Noisy Labels in Earth Observation
• RF-BayesPhysNet: A Bayesian rPPG Uncertainty Estimation Method for Complex Scenarios
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles

Related

• Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization
• On weight and variance uncertainty in neural networks for regression tasks
• Fast Adaptive Weight Noise
• Implicit Weight Uncertainty in Neural Networks
• Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods