Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes

2020/05/17 by Sebastian W. Ober, Laurence Aitchison, Ober, Sebastian W. +1 · 2 citations
Computer Science · Physics and Astronomy · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2005.08140

openalex publication_date 2020/05/17 · openalex created_date 2020/05/21 · openalex updated_date 2026/07/28

Abstract

We consider the optimal approximate posterior over the top-layer weights in a Bayesian neural network for regression, and show that it exhibits strong dependencies on the lower-layer weights. We adapt this result to develop a correlated approximate posterior over the weights at all layers in a Bayesian neural network. We extend this approach to deep Gaussian processes, unifying inference in the two model classes. Our approximate posterior uses learned "global" inducing points, which are defined only at the input layer and propagated through the network to obtain inducing inputs at subsequent layers. By contrast, standard, "local", inducing point methods from the deep Gaussian process literature optimise a separate set of inducing inputs at every layer, and thus do not model correlations across layers. Our method gives state-of-the-art performance for a variational Bayesian method, without data augmentation or tempering, on CIFAR-10 of 86.7%, which is comparable to SGMCMC without tempering but with data augmentation (88% in Wenzel et al. 2020).

Citations

• Adam: A Method for Stochastic Optimization
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Structured Stochastic Variational Inference
• Variational Dropout and the Local Reparameterization Trick
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Convolutional Gaussian Processes
• Deep Neural Networks as Gaussian Processes
• Noisy Natural Gradient as Variational Inference
• Overpruning in Variational Bayesian Neural Networks
• Eigenvalue Corrected Noisy Natural Gradient
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Weight Uncertainty in Neural Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Beyond the Mean-Field: Structured Deep Gaussian Processes Improve the\n Predictive Uncertainties
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• A statistical theory of cold posteriors in deep neural networks
• Deep kernel processes
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Amortising Inference and Meta-Learning Priors in Neural Networks
• PEdger++: Practical Edge Detection via Assembling Cross Information

Related

• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• VIKING: Deep variational inference with stochastic projections
• Sparse Gaussian Processes Revisited: Bayesian Approaches to\n Inducing-Variable Approximations
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian\n Mean Field Posteriors in Bayesian Neural Networks