Posterior Contraction for Sparse Neural Networks in Besov Spaces with Intrinsic Dimensionality

2025/06/23 by Kyeongwon Lee, Lee, Kyeongwon, Lizhen Lin +5
Computer Science · Mathematics · Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical Analysis and Transform Methods #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2506.19144

openalex publication_date 2025/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work establishes that sparse Bayesian neural networks achieve optimal posterior contraction rates over anisotropic Besov spaces and their hierarchical compositions. These structures reflect the intrinsic dimensionality of the underlying function, thereby mitigating the curse of dimensionality. Our analysis shows that Bayesian neural networks equipped with either sparse or continuous shrinkage priors attain the optimal rates which are dependent on the intrinsic dimension of the true structures. Moreover, we show that these priors enable rate adaptation, allowing the posterior to contract at the optimal rate even when the smoothness level of the true function is unknown. The proposed framework accommodates a broad class of functions, including additive and multiplicative Besov functions as special cases. These results advance the theoretical foundations of Bayesian neural networks and provide rigorous justification for their practical effectiveness in high-dimensional, structured estimation problems.

Citations

• Approximation Bounds for Transformer Networks with Application to Regression
• Transformers are Minimax Optimal Nonparametric In-Context Learners
• Posterior and variational inference for deep neural networks with heavy-tailed weights
• Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights
• A Primer on Bayesian Neural Networks: Review and Debates
• Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference
• Asymptotic Properties for Bayesian Neural Network in Besov Space
• Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functions
• Revisiting the Calibration of Modern Neural Networks
• The Intrinsic Dimension of Images and Its Impact on Learning
• Consistent Sparse Deep Learning: Theory and Computation
• Efficient Variational Inference for Sparse Deep Learning with\n Theoretical Guarantee
• Bayesian Deep Learning via Subnetwork Inference
• The art of BART: Minimax optimality over nonhomogeneous smoothness in high dimension
• On the rate of convergence of image classifiers based on convolutional neural networks
• Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Why ReLU networks yield high-confidence predictions far away from the\n training data and how to mitigate the problem
• Fast convergence rates of deep neural networks for classification
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Posterior Concentration for Sparse Deep Learning
• Adaptive Bayesian nonparametric regression using kernel mixture of polynomials with application to partial linear model
• Bayesian fractional posteriors
• Benefits of depth in neural networks
• Representation Benefits of Deep Feedforward Networks
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• The horseshoe estimator for sparse signals

Related

• Asymptotic Properties for Bayesian Neural Network in Besov Space
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• Posterior and variational inference for deep neural networks with heavy-tailed weights
• Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks
• Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights