Practical Bayesian Learning of Neural Networks via Adaptive Optimisation Methods

2018/11/08 by Samuel Kessler, Kessler, Samuel, Arnold Salas +7
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1811.03679

Presented at the ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning

arxiv created 2020/07/20 · arxiv updated 2020/07/21

Abstract

We introduce a novel framework for the estimation of the posterior distribution over the weights of a neural network, based on a new probabilistic interpretation of adaptive optimisation algorithms such as AdaGrad and Adam. We demonstrate the effectiveness of our Bayesian Adam method, Badam, by experimentally showing that the learnt uncertainties correctly relate to the weights' predictive capabilities by weight pruning. We also demonstrate the quality of the derived uncertainty measures by comparing the performance of Badam to standard methods in a Thompson sampling setting for multi-armed bandits, where good uncertainty measures are required for an agent to balance exploration and exploitation.

Related

• Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods
• Approximate Bayesian Optimisation for Neural Networks
• Weight Uncertainty in Neural Networks
• Bayesian Learning for Neural Networks: an algorithmic survey
• Efficacy of Bayesian Neural Networks in Active Learning