Meta Dropout: Learning to Perturb Features for Generalization

2019/05/30 by Haebeom Lee, Lee, Hae Beom, Taewook Nam +5 · 1 citation
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.1905.12914

openalex publication_date 2019/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel regularization method, meta-dropout, which learns to perturb the latent features of training examples for generalization in a meta-learning framework. Specifically, we meta-learn a noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as information bottleneck, manifold mixup, and information dropout.

Citations

• Variational Dropout and the Local Reparameterization Trick
• Auto-Encoding Variational Bayes
• Prototypical Networks for Few-shot Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• On First-Order Meta-Learning Algorithms
• Uncertainty-Aware Attention for Reliable Interpretation and Prediction
• DropBlock: A regularization method for convolutional networks
• Fast Context Adaptation via Meta-Learning
• Adversarial Training and Robustness for Multiple Perturbations
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Cited by

• Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning

Related

• Regularizing Meta-Learning via Gradient Dropout
• Learn to Expect the Unexpected: Probably Approximately Correct Domain\n Generalization
• Invariant Meta Learning for Out-of-Distribution Generalization
• Generating meta-learning tasks to evolve parametric loss for classification learning
• Modeling Generalization in Machine Learning: A Methodological and Computational Study