2020/02/10 by Sebastian Farquhar, Lewis Smith, Farquhar, Sebastian +3 · 2 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Markov Chains and Monte Carlo Methods
paper · pdf · doi:10.48550/arxiv.2002.03704
openalex publication_date 2020/02/10 · openalex created_date 2020/07/16 · openalex updated_date 2026/07/28
We challenge the longstanding assumption that the mean-field approximation for variational inference in Bayesian neural networks is severely restrictive, and show this is not the case in deep networks. We prove several results indicating that deep mean-field variational weight posteriors can induce similar distributions in function-space to those induced by shallower networks with complex weight posteriors. We validate our theoretical contributions empirically, both through examination of the weight posterior using Hamiltonian Monte Carlo in small models and by comparing diagonal- to structured-covariance in large settings. Since complex variational posteriors are often expensive and cumbersome to implement, our results suggest that using mean-field variational inference in a deeper model is both a practical and theoretically justified alternative to structured approximations.