How Important is the Train-Validation Split in Meta-Learning?

2020/10/12 by Yu Bai, Minshuo Chen, Bai, Yu +14 · 12 citations
Computer Science · Engineering · Mathematics · #Algorithm #Artificial intelligence #Centroid #Computer science #Contrast (vision) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Machine learning #Regularization (linguistics) #Sample (material) #Sparse and Compressive Sensing Techniques #Task (project management) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2010.05843

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2020/10/12 · arxiv created 2021/02/09 · arxiv updated 2021/02/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Meta-learning aims to perform fast adaptation on a new task through learning a "prior" from multiple existing tasks. A common practice in meta-learning is to perform a train-validation split (train-val method) where the prior adapts to the task on one split of the data, and the resulting predictor is evaluated on another split. Despite its prevalence, the importance of the train-validation split is not well understood either in theory or in practice, particularly in comparison to the more direct train-train method, which uses all the per-task data for both training and evaluation. We provide a detailed theoretical study on whether and when the train-validation split is helpful in the linear centroid meta-learning problem. In the agnostic case, we show that the expected loss of the train-val method is minimized at the optimal prior for meta testing, and this is not the case for the train-train method in general without structural assumptions on the data. In contrast, in the realizable case where the data are generated from linear models, we show that both the train-val and train-train losses are minimized at the optimal prior in expectation. Further, perhaps surprisingly, our main result shows that the train-train method achieves a strictly better excess loss in this realizable case, even when the regularization parameter and split ratio are optimally tuned for both methods. Our results highlight that sample splitting may not always be preferable, especially when the data is realizable by the model. We validate our theories by experimentally showing that the train-train method can indeed outperform the train-val method, on both simulations and real meta-learning tasks.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Meta-Learning for Semi-Supervised Few-Shot Classification
• On First-Order Meta-Learning Algorithms
• Online Meta-Learning
• Meta-Learning for Low-Resource Neural Machine Translation
• Meta-Learning with Implicit Gradients
• Provable Meta-Learning of Linear Representations
• Few-Shot Learning via Learning the Representation, Provably
• A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
• Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
• On the Theory of Transfer Learning: The Importance of Task Diversity

Cited by

• Task-Level Contrastiveness for Cross-Domain Few-Shot Learning
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• How Fine-Tuning Allows for Effective Meta-Learning
• Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
• How Does the Task Landscape Affect MAML Performance?
• A joint learning framework for fake news detection

Related

• A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• Exploring intra-task relations to improve meta-learning algorithms
• Modeling and Optimization Trade-off in Meta-learning
• On Data Efficiency of Meta-learning