A Practical Bayesian Framework for Backpropagation Networks

1992/05/01 by David J. C. MacKay, David Mackay · 280 citations
Computer Science · #Machine Learning and Algorithms #Bayesian Modeling and Causal Inference #Neural Networks and Applications

paper · doi:10.1162/neco.1992.4.3.448

Abstract

A quantitative and practical Bayesian framework is described for learning of mappings in feedforward networks. The framework makes possible (1) objective comparisons between solutions using alternative network architectures, (2) objective stopping rules for network pruning or growing procedures, (3) objective choice of magnitude and type of weight decay terms or additive regularizers (for penalizing large weights, etc.), (4) a measure of the effective number of well-determined parameters in a model, (5) quantified estimates of the error bars on network parameters and on network output, and (6) objective comparisons with alternative learning and interpolation models such as splines and radial basis functions. The Bayesian "evidence" automatically embodies "Occam's razor," penalizing overflexible and overcomplex models. The Bayesian approach helps detect poor underlying assumptions in learning models. For learning models well matched to a problem, a good correlation between generalization ability and the Bayesian evidence is obtained.

Cited by

• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent
• On the Effect of Regularization on Nonparametric Mean-Variance Regression
• Probabilistic Neuro-Symbolic Reasoning for Sparse Historical Data: A Framework Integrating Bayesian Inference, Causal Models, and Game-Theoretic Allocation
• Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks
• Uncertainty Quantification for Deep Regression using Contextualised Normalizing Flows
• Multi-fidelity Bayesian Optimization Framework for CFD-Based Non-Premixed Burner Design
• Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes
• Noisy Natural Gradient as Variational Inference
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• PAC-Bayes with Backprop
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification
• Repulsive Deep Ensembles are Bayesian
• Invariance of Weight Distributions in Rectified MLPs
• Bayesian Uncertainty Quantification with Anchored Ensembles for Robust EV Power Consumption Prediction
• Learnable Uncertainty under Laplace Approximations
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Predictive Uncertainty Estimation via Prior Networks
• Regularizing Explanations in Bayesian Convolutional Neural Networks
• Riemannian Laplace approximations for Bayesian neural networks
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Information-Based Objective Functions for Active Data Selection
• Neural Network Uncertainty Assessment Using Bayesian Statistics: A Remote Sensing Application
• Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow
• Bayesian Neural Networks at Finite Temperature
• A variational approximate posterior for the deep Wishart process
• Uncertainty-Guided LLM Semantic Augmentation for Heterogeneous Treatment Effect Estimation
• A review of Bayesian neural networks with an application to near infrared spectroscopy
• Learning Structural Weight Uncertainty for Sequential Decision-Making
• On Calibration of Modern Neural Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• Exploratory Modeling for Policy Analysis
• TabImpute: Universal Zero-Shot Imputation for Tabular Data
• A Survey on Bayesian Deep Learning
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Noise Contrastive Priors for Functional Uncertainty
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Analytic Insights into Structure and Rank of Neural Network Hessian Maps
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• SkiNet: A Deep Learning Solution for Skin Lesion Diagnosis with Uncertainty Estimation and Explainability
• Optimality of Sub-network Laplace Approximations: New Results and Methods
• Variational Bayesian Neural Networks via Resolution of Singularities
• Towards Better Performance and More Explainable Uncertainty for 3D Object Detection of Autonomous Vehicles
• Can You Trust This Prediction? Auditing Pointwise Reliability After Learning
• Continual Learning in Recurrent Neural Networks
• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
• The Bayesian Geometry of Transformer Attention
• Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty
• Dropout as a Bayesian Approximation: Appendix
• Bayesian Neural Networks vs. Mixture Density Networks: Theoretical and Empirical Insights for Uncertainty-Aware Nonlinear Modeling
• Sample-efficient and Scalable Exploration in Continuous-Time RL
• Bayesian Speech synthesizers Can Learn from Multiple Teachers
• Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks
• Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation
• Bayesian neural networks with interpretable priors from Mercer kernels
• VIKING: Deep variational inference with stochastic projections
• An uncertainty-aware physics-informed neural network solution for the Black-Scholes equation: a novel framework for option pricing
• What Are Bayesian Neural Network Posteriors Really Like?
• Evidential Turing Processes
• Federated Learning via Meta-Variational Dropout
• A Kernel Framework to Quantify a Model's Local Predictive Uncertainty under Data Distributional Shifts
• No Intelligence Without Statistics: The Invisible Backbone of Artificial Intelligence
• Functional Distribution Networks (FDN)
• Uncertainty-aware data assimilation through variational inference
• Trading with the Devil: Risk and Return in Foundation Model Strategies
• Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization
• A Comparison of the Delta Method and the Bootstrap in Deep Learning Classification
• Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Calibration and Uncertainty Quantification of Bayesian Convolutional Neural Networks for Geophysical Applications
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Structured DropConnect for Uncertainty Inference in Image Classification
• Bayesian Neural Networks for Functional ANOVA model
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• Tracing Uncertainty in Language Model "Reasoning"
• Stochastic Wiring of Cell Types Enhances Fitness by Generating Phenotypic Variability
• Multi-modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation
• Deep Contextual Multi-armed Bandits
• The Sensitivity of Variational Bayesian Neural Network Performance to Hyperparameters
• Weight Uncertainty in Neural Networks
• Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
• Bayesian statistics and modelling
• Approaching Neural Network Uncertainty Realism
• Differentially Private Bayesian Neural Networks on Accuracy, Privacy and Reliability
• A contrastive rule for meta-learning
• Amortising Inference and Meta-Learning Priors in Neural Networks
• Adversarial Phenomenon in the Eyes of Bayesian Deep Learning
• Bayesian Neural Networks versus deep ensembles for uncertainty quantification in machine learning interatomic potentials
• Towards Open-Ended Discovery for Low-Resource NLP
• Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
• Being Bayesian about Categorical Probability
• Accuracy-Reliability Cost Function for Empirical Variance Estimation
• RCoNet: Deformable Mutual Information Maximization and High-order Uncertainty-aware Learning for Robust COVID-19 Detection
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Two Birds with One Stone: Enhancing Uncertainty Quantification and Interpretability with Graph Functional Neural Process
• Stochastic Neural Network with Kronecker Flow
• UnLoc: Leveraging Depth Uncertainties for Floorplan Localization
• Action and Perception as Divergence Minimization
• Kalman Bayesian Transformer
• A Parsimonious Tour of Bayesian Model Uncertainty
• Learning Curves for Gaussian Process Regression: Approximations and Bounds
• Conditional Density Estimation with Bayesian Normalising Flows
• Stochastic Aggregation in Graph Neural Networks
• Adaptive, Distribution-Free Prediction Intervals for Deep Networks
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• TyXe: Pyro-based Bayesian neural nets for Pytorch
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction
• Comparison of Deep Neural Networks and Deep Hierarchical Models for Spatio-Temporal Data
• Sampling-based Bayesian Inference with gradient uncertainty
• Understanding Classifier Mistakes with Generative Models
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Reliable training and estimation of variance networks
• Interactive Medical Image Segmentation with Self-Adaptive Confidence Calibration
• Uncertainty Tube Visualization of Particle Trajectories
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• Optimization-based Online Conformal Prediction for Multi-step Forecasting
• Understanding Uncertainty in Bayesian Deep Learning
• Calibrated Reliable Regression using Maximum Mean Discrepancy
• Integrating Scientific Knowledge with Machine Learning for Engineering and Environmental Systems
• PEdger++: Practical Edge Detection via Assembling Cross Information
• Nonparametric learning of stochastic differential equations from sparse and noisy data
• Non-Parametric Calibration for Classification
• Deep kernel processes
• Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Two Steps Feature Selection and Neural Network Classification for the TREC-8 Routing
• A Simple and Effective Method for Uncertainty Quantification and OOD Detection
• Semantic-Aware Gaussian Process Calibration with Structured Layerwise Kernels for Deep Neural Networks
• Bayesian Neural Network Surrogates for Bayesian Optimization of Carbon Capture and Storage Operations
• Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence
• Adversarial Distillation of Bayesian Neural Network Posteriors
• A Unifying Bayesian View of Continual Learning
• Constructive algorithms for structure learning in feedforward neural networks for regression problems
• A new methodology of extraction, optimization and application of crisp and fuzzy logical rules
• On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
• DeepHelp: Deep Learning for Shout Crisis Text Conversations
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Information-Theoretic Local Minima Characterization and Regularization
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Role of calibration in uncertainty-based referral for deep learning
• Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks
• laplax -- Laplace Approximations with JAX
• Control Neuronal por Modelo Inverso de un Servosistema Usando Algoritmos de Aprendizaje Levenberg-Marquardt y Bayesiano
• Applying MDL to Learning Best Model Granularity
• Machine Learning Advances for Time Series Forecasting
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• A Latent Gaussian Process Model with Application to Monitoring Clinical Trials
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• An Artificial Neural Network for Gait Analysis to Estimate Blood Alcohol Content Level
• Nonparametric Neural Networks
• Being a Bit Frequentist Improves Bayesian Neural Networks
• Task Agnostic Continual Learning Using Online Variational Bayes
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning
• Towards Error Measures which Influence a Learners Inductive Bias to the Ground Truth
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models
• CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk
• Notes on the Behavior of MC Dropout
• L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation
• Decomposing Prediction Mechanisms for In-Context Recall
• Posterior Meta-Replay for Continual Learning
• Uncertainty-guided Continual Learning with Bayesian Neural Networks
• Reversible Jump MCMC Simulated Annealing for Neural Networks
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• FedRef: Communication-Efficient Bayesian Fine-Tuning using a Reference Model
• rodeo: Probabilistic Methods of Parameter Inference for Ordinary Differential Equations
• Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry
• InvZW: Invariant Feature Learning via Noise-Adversarial Training for Robust Image Zero-Watermarking
• On Equivariant Model Selection through the Lens of Uncertainty
• Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Probabilistic Residual Learning for Aleatoric Uncertainty in Image\n Restoration
• An Uncertainty-Aware Dynamic Decision Framework for Progressive Multi-Omics Integration in Classification Tasks
• Out-of-Distribution Example Detection in Deep Neural Networks using Distance to Modelled Embedding
• Uncertainty Estimation by Human Perception versus Neural Models
• Improving LoRA with Variational Learning
• Deep learning inference with the Event Horizon Telescope II. The Zingularity framework for Bayesian artificial neural networks
• Single Shot MC Dropout Approximation
• Sequential Anomaly Detection using Inverse Reinforcement Learning
• Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'
• Active Learning in CNNs via Expected Improvement Maximization
• Variational Inference in Nonconjugate Models
• Probabilistic Safety for Bayesian Neural Networks
• Regularising Deep Networks with Deep Generative Models
• Probabilistic Model-Agnostic Meta-Learning
• Bayesian deep neural networks for low-cost neurophysiological markers of Alzheimer's disease severity
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• Beyond Point Estimate: Inferring Ensemble Prediction Variation from Neuron Activation Strength in Recommender Systems
• Prior Activation Distribution (PAD): A Versatile Representation to Utilize DNN Hidden Units
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Early Stopping is Nonparametric Variational Inference
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
• Uncertainty estimation under model misspecification in neural network regression
• Variational Bayesian Quantization
• Principled Input-Output-Conditioned Post-Hoc Uncertainty Estimation for Regression Networks
• Towards Robust Evaluations of Continual Learning
• Using Ensemble Diffusion to Estimate Uncertainty for End-to-End Autonomous Driving
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Position: Epistemic uncertainty estimation methods are fundamentally incomplete
• Bayesian Field Theory: Nonparametric Approaches to Density Estimation, Regression, Classification, and Inverse Quantum Problems
• Walking the Weight Manifold: a Topological Approach to Conditioning Inspired by Neuromodulation
• Artificial Neural Network Modeling of Forest Tree Growth
• CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment
• Optimizing Data Augmentation through Bayesian Model Selection
• Variational Deep Learning via Implicit Regularization
• On Machine Learning and Structure for Mobile Robots
• Sparfels: Fast Reconstruction from Sparse Unposed Imagery
• MoMBS: Mixed-order minibatch sampling enhances model training from diverse-quality images
• Exemplar-Free Continual Learning for State Space Models
• A Bayesian Approach to Invariant Deep Neural Networks
• Predictive Uncertainty Quantification with Compound Density Networks
• Decision Theoretic Bootstrapping
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• High Dimensional Level Set Estimation with Bayesian Neural Network
• ProbAct: A Probabilistic Activation Function for Deep Neural Networks
• Constrained Co-Design for Photonic Bayesian Neural Networks
• Confidence Modeling for Neural Semantic Parsing
• Predicting mosquito flight behavior using Bayesian dynamical systems learning
• Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensembles
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Geometrically Enriched Latent Spaces
• Are vision language models robust to uncertain inputs?
• Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction tasks
• Recursive Gaussian Processes and the Bayesian Brain
• Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models
• Neural Variational Gradient Descent
• Eigenvalue Corrected Noisy Natural Gradient
• Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks
• Learning Neural Networks with Adaptive Regularization
• Sparse Latent Factor Forecaster (SLFF) with Iterative Inference for Transparent Multi-Horizon Commodity Futures Prediction
• Bayesian Neural Network Ensembles
• GeneDisco: A Benchmark for Experimental Design in Drug Discovery
• Inferring Parameters and Structure of Latent Variable Models by Variational Bayes
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Loss-Calibrated Approximate Inference in Bayesian Neural Networks
• Monitoring multimode processes: a modified PCA algorithm with continual learning ability
• Gaussian process modelling with Gaussian mixture likelihood
• Understanding Softmax Confidence and Uncertainty
• A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
• RBUE: A ReLU-Based Uncertainty Estimation Method of Deep Neural Networks
• Continual Learning in Human Activity Recognition: an Empirical Analysis of Regularization
• Deepening Lee-Carter for longevity projections with uncertainty estimation
• Continual Domain-Tuning for Pretrained Language Models
• Why have a Unified Predictive Uncertainty? Disentangling it using Deep Split Ensembles
• Self-Aware Object Detection via Degradation Manifolds
• Scalable Uncertainty Quantification for Extreme Weather Forecasting via Empirical Neural Tangent Kernels
• Application of Neural Networks in Diagnosing Cancer Disease using Demographic Data
• Significant wave height record extension by neural networks and reanalysis wind data
• Laplace-Bridged Randomized Smoothing for Fast Certified Robustness
• It’s all in your head—naturalness arguments do not require aleatoric uncertainty
• Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning
• Two-photon exchange effect studied with neural networks
• Bayesian Networks for Data Mining
• Overcoming catastrophic forgetting in neural networks
• Stochastic estimation of dynamical variables
• Learning free energy landscapes using artificial neural networks
• Mixed Variational Inference
• Confidence Sequences for Generalized Linear Models via Regret Analysis
• Bayesian neural networks at scale: a performance analysis and pruning study
• Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
• Neural network parameterizations of electromagnetic nucleon form-factors
• Invariance-embedded Machine Learning Sub-grid-scale Stress Models for Meso-scale Hurricane Boundary Layer Flow Simulation I: Model Development and a priori Studies
• Bayesian Neural Network Priors Revisited
• Variational inference as an alternative to MCMC for parameter estimation and model selection
• Distributed Variational Bayesian Algorithms Over Sensor Networks
• Learnable Bernoulli Dropout for Bayesian Deep Learning
• Epistemic Uncertainty-aware Recommendation Systems via Bayesian Deep Ensemble Learning
• Estimation of Accurate and Calibrated Uncertainties in Deterministic models

Related

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Natural-Parameter Networks: A Class of Probabilistic Neural Networks
• Bayesian Neural Networks
• A Tutorial on Learning With Bayesian Networks
• The Case for Bayesian Deep Learning