Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation

2019/09/06 by Pitas, Konstantinos
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1909.03009

Abstract

Explaining how overparametrized neural networks simultaneously achieve low risk and zero empirical risk on benchmark datasets is an open problem. PAC-Bayes bounds optimized using variational inference (VI) have been recently proposed as a promising direction in obtaining non-vacuous bounds. We show empirically that this approach gives negligible gains when modeling the posterior as a Gaussian with diagonal covariance--known as the mean-field approximation. We investigate common explanations, such as the failure of VI due to problems in optimization or choosing a suboptimal prior. Our results suggest that investigating richer posteriors is the most promising direction forward.

Related

• On the detrimental effect of invariances in the likelihood for variational inference
• Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations
• Stability-based Generalization Bounds for Variational Inference
• Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data
• VIKING: Deep variational inference with stochastic projections