Radial and Directional Posteriors for Bayesian Neural Networks

2019/02/07 by ChangYong Oh, Kamil Adamczewski, Oh, Changyong +3
Computer Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1902.02603

openalex publication_date 2019/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a new variational family for Bayesian neural networks. We decompose the variational posterior into two components, where the radial component captures the strength of each neuron in terms of its magnitude; while the directional component captures the statistical dependencies among the weight parameters. The dependencies learned via the directional density provide better modeling performance compared to the widely-used Gaussian mean-field-type variational family. In addition, the strength of input and output neurons learned via the radial density provides a structured way to compress neural networks. Indeed, experiments show that our variational family improves predictive performance and yields compressed networks simultaneously.

Citations

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Auto-Encoding Variational Bayes
• Structured and Efficient Variational Deep Learning with Matrix Gaussian\n Posteriors
• Categorical Reparameterization with Gumbel-Softmax
• Variational Dropout Sparsifies Deep Neural Networks
• REBAR: Low-variance, unbiased gradient estimates for discrete latent\n variable models
• Weight Uncertainty in Neural Networks
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Related

• Variational Bayesian Neural Networks via Resolution of Singularities
• Variational Bayesian Neural Networks via Resolution of Singularities
• Bayesian Learning for Neural Networks: an algorithmic survey
• Precise Bayesian Neural Networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes