An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their\n Asymptotic Overconfidence

2020/10/06 by Agustinus Kristiadi, Matthias Hein, Kristiadi, Agustinus +3 · 1 citation
Computer Science · Engineering · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Fault Detection and Control Systems #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2010.02709

openalex publication_date 2020/10/06 · openalex created_date 2022/07/17 · openalex updated_date 2026/07/28

Abstract

A Bayesian treatment can mitigate overconfidence in ReLU nets around the\ntraining data. But far away from them, ReLU Bayesian neural networks (BNNs) can\nstill underestimate uncertainty and thus be asymptotically overconfident. This\nissue arises since the output variance of a BNN with finitely many features is\nquadratic in the distance from the data region. Meanwhile, Bayesian linear\nmodels with ReLU features converge, in the infinite-width limit, to a\nparticular Gaussian process (GP) with a variance that grows cubically so that\nno asymptotic overconfidence can occur. While this may seem of mostly\ntheoretical interest, in this work, we show that it can be used in practice to\nthe benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via\nthe GP and show that the resulting model is asymptotically maximally uncertain\nfar away from the data while the BNNs' predictive power is unaffected near the\ndata. Although the resulting model approximates a full GP posterior, thanks to\nits structure, it can be applied \post-hoc to any pre-trained ReLU BNN at\na low cost.\n

Citations

• Scalable Variational Gaussian Process Classification
• Gradient-based learning applied to document recognition
• Stochastic Variational Deep Kernel Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Deep Anomaly Detection with Outlier Exposure
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Cited by

• Being a Bit Frequentist Improves Bayesian Neural Networks

Related

• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Why ReLU networks yield high-confidence predictions far away from the\n training data and how to mitigate the problem
• Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Cases
• Noisy Interpolation Learning with Shallow Univariate ReLU Networks
• Student-t processes as infinite-width limits of posterior Bayesian neural networks