2022/02/08 by Meet P. Vadera, Adam D. Cobb, Vadera, Meet P. +5
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2202.03770
Preprint. Work in progress
arxiv created 2022/02/08 · openalex publication_date 2022/02/08 · arxiv updated 2022/02/09 · openalex created_date 2022/04/03 · openalex updated_date 2026/07/28
Bayesian methods hold significant promise for improving the uncertainty quantification ability and robustness of deep neural network models. Recent research has seen the investigation of a number of approximate Bayesian inference methods for deep neural networks, building on both the variational Bayesian and Markov chain Monte Carlo (MCMC) frameworks. A fundamental issue with MCMC methods is that the improvements they enable are obtained at the expense of increased computation time and model storage costs. In this paper, we investigate the potential of sparse network structures to flexibly trade-off model storage costs and inference run time against predictive performance and uncertainty quantification ability. We use stochastic gradient MCMC methods as the core Bayesian inference method and consider a variety of approaches for selecting sparse network structures. Surprisingly, our results show that certain classes of randomly selected substructures can perform as well as substructures derived from state-of-the-art iterative pruning methods while drastically reducing model training times.