DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a\n Trained Classifier

2019/12/26 by Sravanti Addepalli, Gaurav Kumar Nayak, Addepalli, Sravanti +5 · 1 citation
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.1912.11960

openalex publication_date 2019/12/26 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28

Abstract

In this era of digital information explosion, an abundance of data from\nnumerous modalities is being generated as well as archived everyday. However,\nmost problems associated with training Deep Neural Networks still revolve\naround lack of data that is rich enough for a given task. Data is required not\nonly for training an initial model, but also for future learning tasks such as\nModel Compression and Incremental Learning. A diverse dataset may be used for\ntraining an initial model, but it may not be feasible to store it throughout\nthe product life cycle due to data privacy issues or memory constraints. We\npropose to bridge the gap between the abundance of available data and lack of\nrelevant data, for the future learning tasks of a given trained network. We use\nthe available data, that may be an imbalanced subset of the original training\ndataset, or a related domain dataset, to retrieve representative samples from a\ntrained classifier, using a novel Data-enriching GAN (DeGAN) framework. We\ndemonstrate that data from a related domain can be leveraged to achieve\nstate-of-the-art performance for the tasks of Data-free Knowledge Distillation\nand Incremental Learning on benchmark datasets. We further demonstrate that our\nproposed framework can enrich any data, even from unrelated domains, to make it\nmore useful for the future learning tasks of a given network.\n

Citations

• Distilling the Knowledge in a Neural Network
• Gradient-based learning applied to document recognition
• Blind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
• Improved Techniques for Training GANs
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Continual Learning with Deep Generative Replay
• Statistics of Deep Generated Images
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Data-Free Knowledge Distillation for Deep Neural Networks
• Learning from Higher-Layer Feature Visualizations
• Zero-Shot Knowledge Distillation in Deep Networks
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments

Related

• MineGAN: effective knowledge transfer from GANs to target domains with few images
• A Data-Centric Approach for Training Deep Neural Networks with Less Data
• MineGAN++: Mining Generative Models for Efficient Knowledge Transfer to Limited Data Domains
• Training Generative Adversarial Networks with Limited Data
• Memorization Precedes Generation: Learning Unsupervised GANs with Memory Networks