Noisy Natural Gradient as Variational Inference

2017/12/06 by Guodong Zhang, Shengyang Sun, Zhang, Guodong +5 · 17 citations
Computer Science · Mathematics · #Gaussian Processes and Bayesian Inference #Machine Learning and Algorithms #Machine Learning and Data Classification #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1712.02390

arxiv created 2018/02/26 · arxiv updated 2018/02/27

Abstract

Variational Bayesian neural nets combine the flexibility of deep learning with Bayesian uncertainty estimation. Unfortunately, there is a tradeoff between cheap but simple variational families (e.g.~fully factorized) or expensive and complicated inference procedures. We show that natural gradient ascent with adaptive weight noise implicitly fits a variational posterior to maximize the evidence lower bound (ELBO). This insight allows us to train full-covariance, fully factorized, or matrix-variate Gaussian variational posteriors using noisy versions of natural gradient, Adam, and K-FAC, respectively, making it possible to scale up to modern-size ConvNets. On standard regression benchmarks, our noisy K-FAC algorithm makes better predictions and matches Hamiltonian Monte Carlo's predictive variances better than existing methods. Its improved uncertainty estimates lead to more efficient exploration in active learning, and intrinsic motivation for reinforcement learning.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Practical Bayesian Framework for Backpropagation Networks

Cited by

• Natural gradient descent for improving variational inference based classification of radio galaxies
• A statistical theory of cold posteriors in deep neural networks
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Natural Gradient VI: Guarantees for Non-Conjugate Models
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• The Functional Neural Process
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Better Training Data Attribution via Better Inverse Hessian-Vector Products
• Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators
• Task Agnostic Continual Learning Using Online Variational Bayes
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Variational Deep Learning via Implicit Regularization
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• Foundations of Top-k Decoding For Language Models
• Eigenvalue Corrected Noisy Natural Gradient
• Learning Neural Networks with Adaptive Regularization

Related

• Eigenvalue Corrected Noisy Natural Gradient
• The Variational Predictive Natural Gradient
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Deterministic Variational Inference for Robust Bayesian Neural Networks
• Natural Gradient Variational Inference with Gaussian Mixture Models