Variational Resampling Based Assessment of Deep Neural Networks under Distribution Shift

2019/06/07 by Xudong Sun, Sun, Xudong, Alexej Gossmann +5
Computer Science · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1906.02972

openalex publication_date 2019/06/07 · openalex created_date 2019/08/13 · openalex updated_date 2026/08/04

Abstract

A novel variational inference based resampling framework is proposed to evaluate the robustness and generalization capability of deep learning models with respect to distribution shift. We use Auto Encoding Variational Bayes to find a latent representation of the data, on which a Variational Gaussian Mixture Model is applied to deliberately create distribution shift by dividing the dataset into different clusters. Wasserstein distance is used to characterize the extent of distribution shift between the generated data splits. We compare several popular Convolutional Neural Network (CNN) architectures and Bayesian CNN models for image classification on the Fashion-MNIST dataset, to assess their robustness and generalization behavior under the deliberately created distribution shift, as well as under random Cross Validation. Our method of creating artificial domain splits of a single dataset can also be used to establish novel model selection criteria and assessment tools in machine learning, as well as benchmark methods for domain adaptation and domain generalization approaches.

Citations

• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Auto-Encoding Variational Bayes
• Gradient-based learning applied to document recognition
• ImageNet classification with deep convolutional neural networks
• Variational Dropout Sparsifies Deep Neural Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization
• Weight Uncertainty in Neural Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Benchmark for filter methods for feature selection in high-dimensional classification data

Related

• Variational Inference to Measure Model Uncertainty in Deep Neural Networks
• Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift
• Variational Auto-encoded Deep Gaussian Processes
• VIKING: Deep variational inference with stochastic projections
• Variational Bayesian Bow tie Neural Networks with Shrinkage