Stochastic Neural Network with Kronecker Flow

2019/06/10 by Chin-Wei Huang, Huang, Chin-Wei, Pascal Vincent +8
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.1906.04282

openalex publication_date 2019/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and better than the baselines.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural\n Networks
• Variational Dropout and the Local Reparameterization Trick
• A Stochastic Approximation Method
• A Practical Bayesian Framework for Backpropagation Networks
• Gradient-based learning applied to document recognition
• Stronger generalization bounds for deep nets via a compression approach
• Block Neural Autoregressive Flow
• Conditional Image Generation with PixelCNN Decoders
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• PAC-Bayes with Backprop
• Learning Multiple Layers of Features from Tiny Images

Related

• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Stochastic Normalizing Flows
• Variational Inference on the Final-Layer Output of Neural Networks
• Precise Bayesian Neural Networks
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian\n Mean Field Posteriors in Bayesian Neural Networks