2020/03/02 by Marius Hobbhahn, Hobbhahn, Marius, Agustinus Kristiadi +3 · 3 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML)
paper · pdf · doi:10.48550/arxiv.2003.01227
openalex publication_date 2020/03/02 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
In Bayesian Deep Learning, distributions over the output of classification\nneural networks are often approximated by first constructing a Gaussian\ndistribution over the weights, then sampling from it to receive a distribution\nover the softmax outputs. This is costly. We reconsider old work (Laplace\nBridge) to construct a Dirichlet approximation of this softmax output\ndistribution, which yields an analytic map between Gaussian distributions in\nlogit space and Dirichlet distributions (the conjugate prior to the Categorical\ndistribution) in the output space. Importantly, the vanilla Laplace Bridge\ncomes with certain limitations. We analyze those and suggest a simple solution\nthat compares favorably to other commonly used estimates of the\nsoftmax-Gaussian integral. We demonstrate that the resulting Dirichlet\ndistribution has multiple advantages, in particular, more efficient computation\nof the uncertainty estimate and scaling to large datasets and networks like\nImageNet and DenseNet. We further demonstrate the usefulness of this Dirichlet\napproximation by using it to construct a lightweight uncertainty-aware output\nranking for ImageNet.\n