Weight Uncertainty in Neural Networks

2015/05/20 by Charles Blundell, Blundell, Charles, Julien Cornebise +5 · 100 citations
Computer Science · Decision Sciences · #Advanced Bandit Algorithms Research #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.1505.05424

openalex publication_date 2015/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Auto-Encoding Variational Bayes
• A Practical Bayesian Framework for Backpropagation Networks

Cited by

• DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
• Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
• Am I Confused or Is This Confusing?: Deep Ensembles for ENSO Uncertainty Quantification
• Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertainty
• Transfer Learning for Analysis of Collective and Non-Collective Thomson Scattering Spectra
• Toward Ethical AI Through Bayesian Uncertainty in Neural Question Answering
• Unreliable Uncertainty Estimates with Monte Carlo Dropout
• Selective Conformal Risk Control
• PIS: A Generalized Physical Inversion Solver for Arbitrary Sparse Observations via Set Conditioned Flow Matching
• Uncertainty-Preserving QBNNs: Multi-Level Quantization of SVI-Based Bayesian Neural Networks for Image Classification
• Minimization of Functions on Dually Flat Spaces Using Geodesic Descent Based on Dual Connections
• Bayesian Physics-Informed Neural Networks for Inverse Problems (BPINN-IP): Application in Infrared Image Processing
• Uncertainty Reasoning with Photonic Bayesian Machines
• Probabilistic Neuro-Symbolic Reasoning for Sparse Historical Data: A Framework Integrating Bayesian Inference, Causal Models, and Game-Theoretic Allocation
• Nanosecond-Scale Proton Emission from Triaxially Deformed Lu-148 Predicted with High Accuracy Qp Value via Novel Bayesian Evaluation
• Accelerated Execution of Bayesian Neural Networks using a Single Probabilistic Forward Pass and Code Generation
• PaTAS: A Parallel System for Trust Propagation in Neural Networks Using Subjective Logic
• Training-Free Active Learning Framework in Materials Science with Large Language Models
• Natural gradient descent for improving variational inference based classification of radio galaxies
• Bayesian--AI Fusion for Epidemiological Decision Making: Calibrated Risk, Honest Uncertainty, and Hyperparameter Intelligence
• Uncertainty Calibration of Multi-Label Bird Sound Classifiers
• Uncertainty Guided Online Ensemble for Non-stationary Data Streams in Fusion Science
• Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
• Uncertainty-Guided LLM Semantic Augmentation for Heterogeneous Treatment Effect Estimation
• The Bayesian Geometry of Transformer Attention
• FruitProm: Probabilistic Maturity Estimation and Detection of Fruits and Vegetables
• VIKING: Deep variational inference with stochastic projections
• Surrogate-based quantification of policy uncertainty in generative flow networks
• Functional Distribution Networks (FDN)
• Bayesian Topological Convolutional Neural Nets
• A Systematic Literature Review of Machine Learning Techniques for Observational Constraints in Cosmology
• Robust Classification of Oral Cancer with Limited Training Data
• Bayesian Surrogates for Risk-Aware Pre-Assessment of Aging Bridge Portfolios
• Uncertainty-Aware Deep Learning for Wildfire Danger Forecasting
• Tracing Uncertainty in Language Model "Reasoning"
• Learnable Conformal Prediction with Context-Aware Nonconformity Functions for Robotic Planning and Perception
• Multi-modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation
• Approaching Neural Network Uncertainty Realism
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Uncertainty Estimation in Deep Neural Networks for Point Cloud\n Segmentation in Factory Planning
• Confidence-Aware Routing for Large Language Model Reliability Enhancement: A Multi-Signal Approach to Pre-Generation Hallucination Mitigation
• Towards Open-Ended Discovery for Low-Resource NLP
• A Unified Particle-Optimization Framework for Scalable Bayesian Sampling
• AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
• Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
• Information Geometry of Variational Bayes
• Being Bayesian about Categorical Probability
• A Conformal Prediction Framework for Uncertainty Quantification in Physics-Informed Neural Networks
• RCoNet: Deformable Mutual Information Maximization and High-order Uncertainty-aware Learning for Robust COVID-19 Detection
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Unifying Variational Inference and PAC-Bayes for Supervised Learning that Scales
• MC²RAM: Markov Chain Monte Carlo Sampling in SRAM for Fast Bayesian\n Inference
• Action and Perception as Divergence Minimization
• Ensemble Distribution Distillation for Self-Supervised Human Activity Recognition
• Disentangling the Gauss-Newton Method and Approximate Inference for Neural Networks
• Sparsifying Neural Network Connections for Face Recognition
• Peer-to-peer Federated Learning on Graphs
• Sparse Uncertainty Representation in Deep Learning with Inducing Weights
• A probabilistic approach to tomography and adjoint state methods, with an application to full waveform inversion in medical ultrasound
• Data Uncertainty Learning in Face Recognition
• Uncertainty based Class Activation Maps for Visual Question Answering
• Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentation
• Compressing Neural Networks using the Variational Information Bottleneck
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time\n Augmentation
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• TRADI: Tracking deep neural network weight distributions for uncertainty\n estimation
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Masked Contrastive Learning for Anomaly Detection
• Diversity inducing Information Bottleneck in Model Ensembles
• Sampling-based Bayesian Inference with gradient uncertainty
• Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation
• Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Estimating Model Uncertainty of Neural Networks in Sparse Information\n Form
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Improving Local Identifiability in Probabilistic Box Embeddings
• HierCVAE: Hierarchical Attention-Driven Conditional Variational Autoencoders for Multi-Scale Temporal Modeling
• Approximate Bayesian Optimisation for Neural Networks
• Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
• NP-ODE: Neural Process Aided Ordinary Differential Equations for Uncertainty Quantification of Finite Element Analysis
• Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Uncertainty Tube Visualization of Particle Trajectories
• BLIPs: Bayesian Learned Interatomic Potentials
• Understanding Uncertainty in Bayesian Deep Learning
• Verifying the Causes of Adversarial Examples
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic\n Convergence Theory
• Deep density networks and uncertainty in recommender systems
• Improving Back-Translation with Uncertainty-based Confidence Estimation
• Routing Networks with Co-training for Continual Learning
• On Hallucination and Predictive Uncertainty in Conditional Language Generation
• A Generative Model for Sampling High-Performance and Diverse Weights for\n Neural Networks
• Bayesian Distributional Policy Gradients
• Stochastic Maximum Likelihood Optimization via Hypernetworks
• Explainable Image Classification with Reduced Overconfidence for Tissue Characterisation
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• A Novel Regression Loss for Non-Parametric Uncertainty Optimization

Related

• Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization
• On weight and variance uncertainty in neural networks for regression tasks
• Fast Adaptive Weight Noise
• Implicit Weight Uncertainty in Neural Networks
• Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods