Being a Bit Frequentist Improves Bayesian Neural Networks

2021/06/18 by Agustinus Kristiadi, Matthias Hein, Kristiadi, Agustinus +3 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2106.10065

openalex publication_date 2021/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this "OOD training" technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.

Citations

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• A Practical Bayesian Framework for Backpropagation Networks
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Evidential Deep Learning to Quantify Classification Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their\n Asymptotic Overconfidence

Cited by

• VIKING: Deep variational inference with stochastic projections
• Uncertainty-Aware Graph Neural Networks: A Multi-Hop Evidence Fusion Approach

Related

• Out-of-Distribution Detection from Small Training Sets using Bayesian Neural Network Classifiers
• Are Bayesian neural networks intrinsically good at out-of-distribution detection?
• On out-of-distribution detection with Bayesian neural networks
• Be Bayesian by Attachments to Catch More Uncertainty
• Known Unknowns: Uncertainty Quality in Bayesian Neural Networks