Model Selection in Bayesian Neural Networks via Horseshoe Priors

2017/05/29 by Soumya Ghosh, Finale Doshi-Velez, Ghosh, Soumya +2 · 4 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (stat.ML) #Machine Learning and Algorithms #stat.ML

paper · pdf · doi:10.48550/arxiv.1705.10388

arxiv created 2017/05/29 · openalex publication_date 2017/05/29 · arxiv updated 2017/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. In this work, we apply a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

Citations

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• A Practical Bayesian Framework for Backpropagation Networks
• Learning Structured Sparsity in Deep Neural Networks
• Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization
• Weight Uncertainty in Neural Networks
• Deep Bayesian Active Learning with Image Data
• Auto-Sizing Neural Networks: With Applications to n-gram Language Models
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Cited by

• Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance
• Feature Preserving Shrinkage on Bayesian Neural Networks via the R2D2 Prior
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Joint Model and Data Sparsification via the Marginal Likelihood

Related

• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• Efficient Model Compression for Bayesian Neural Networks
• A Primer on Bayesian Neural Networks: Review and Debates
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details