Learnable Uncertainty under Laplace Approximations

2020/10/06 by Agustinus Kristiadi, Matthias Hein, Kristiadi, Agustinus +3 · 5 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2010.02720

openalex publication_date 2020/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Laplace approximations are classic, computationally lightweight means for constructing Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop a formalism to explicitly "train" the uncertainty in a decoupled way to the prediction itself. To this end, we introduce uncertainty units for Laplace-approximated networks: Hidden units associated with a particular weight structure that can be added to any pre-trained, point-estimated network. Due to their weights, these units are inactive -- they do not affect the predictions. But their presence changes the geometry (in particular the Hessian) of the loss landscape, thereby affecting the network's uncertainty estimates under a Laplace approximation. We show that such units can be trained via an uncertainty-aware objective, improving standard Laplace approximations' performance in various uncertainty quantification tasks.

Citations

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural\n Networks
• Natural Gradient Works Efficiently in Learning
• VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
• A Practical Bayesian Framework for Backpropagation Networks
• Deep Anomaly Detection with Outlier Exposure

Cited by

• CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regression
• Kalman Bayesian Transformer
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• Monte Carlo Functional Regularisation for Continual Learning
• Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?

Related

• Variation Due to Regularization Tractably Recovers Bayesian Deep Learning
• Projected BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights
• Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks
• Measuring Uncertainty through Bayesian Learning of Deep Neural Network Structure
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks