Risk bounds for aggregated shallow neural networks using Gaussian prior

2021/12/21 by Laura Tinsi, Tinsi, Laura, Arnak S. Dalalyan +1 · 1 citation
Computer Science · Mathematics · #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Machine Learning and Algorithms #Statistical Methods and Inference #Statistics Theory (math.ST) #math.ST #stat.TH

paper · pdf · doi:10.48550/arxiv.2112.11086

openalex publication_date 2021/12/21 · arxiv created 2022/02/02 · arxiv updated 2022/02/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Analysing statistical properties of neural networks is a central topic in statistics and machine learning. However, most results in the literature focus on the properties of the neural network minimizing the training error. The goal of this paper is to consider aggregated neural networks using a Gaussian prior. The departure point of our approach is an arbitrary aggregate satisfying the PAC-Bayesian inequality. The main contribution is a precise nonasymptotic assessment of the estimation error appearing in the PAC-Bayes bound. We also review available bounds on the error of approximating a function by a neural network. Combining bounds on estimation and approximation errors, we establish risk bounds that are sharp enough to lead to minimax rates of estimation over Sobolev smoothness classes.

Cited by

• Kullback-Leibler excess risk bounds for exponential weighted aggregation in Generalized linear models

Related

• PAC-Bayesian risk bounds for fully connected deep neural network with Gaussian priors
• On the Importance of Gradient Norm in PAC-Bayesian Bounds
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• PAC-Bayesian Learning of Aggregated Binary Activated Neural Networks with Probabilities over Representations
• Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance