Understanding Priors in Bayesian Neural Networks at the Unit Level

2018/10/11 by Mariia Vladimirova, Jakob Verbeek, Vladimirova, Mariia +5 · 1 citation
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1810.05193

openalex publication_date 2018/10/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments.

Cited by

• Convergence Rates of Variational Inference in Sparse Deep Learning

Related

• Bayesian neural network unit priors and generalized Weibull-tail property
• All You Need is a Good Functional Prior for Bayesian Deep Learning
• Explicitly Bayesian Regularizations in Deep Learning
• Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
• Bayesian Neural Network Priors Revisited