Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation

2018/05/19 by Haussmann, Manuel, Hamprecht, Fred A., Kandemir, Melih
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1805.07654

Abstract

We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) introducing a separate path that decomposes the neural net expectation from its variance. We demonstrate formally that introducing separate latent binary variables to the activations allows representing the neural network likelihood as a chain of linear operations. Performing variational inference on this construction enables a sampling-free computation of the evidence lower bound which is a more effective approximation than the widely applied Monte Carlo sampling and CLT related techniques. We evaluate the model on a range of regression and classification tasks against BNN inference alternatives, showing competitive or improved performance over the current state-of-the-art.

Related

• Sampling-free Variational Inference for Neural Networks with\n Multiplicative Activation Noise
• Deterministic Variational Inference for Robust Bayesian Neural Networks
• Few-sample Variational Inference of Bayesian Neural Networks with Arbitrary Nonlinearities
• Central Limit Theorem for Bayesian Neural Network trained with Variational Inference
• Variational Laplace for Bayesian neural networks