Provable Meta-Learning of Linear Representations

2020/02/26 by Nilesh Tripuraneni, Chi Jin, Tripuraneni, Nilesh +3 · 16 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Machine Learning and Data Classification #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2002.11684

Abstract

Meta-learning, or learning-to-learn, seeks to design algorithms that can utilize previous experience to rapidly learn new skills or adapt to new environments. Representation learning -- a key tool for performing meta-learning -- learns a data representation that can transfer knowledge across multiple tasks, which is essential in regimes where data is scarce. Despite a recent surge of interest in the practice of meta-learning, the theoretical underpinnings of meta-learning algorithms are lacking, especially in the context of learning transferable representations. In this paper, we focus on the problem of multi-task linear regression -- in which multiple linear regression models share a common, low-dimensional linear representation. Here, we provide provably fast, sample-efficient algorithms to address the dual challenges of (1) learning a common set of features from multiple, related tasks, and (2) transferring this knowledge to new, unseen tasks. Both are central to the general problem of meta-learning. Finally, we complement these results by providing information-theoretic lower bounds on the sample complexity of learning these linear features.

Citations

• On the limited memory BFGS method for large scale optimization
• A Simpler Approach to Matrix Completion
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Few-Shot Learning via Learning the Representation, Provably

Cited by

• Diffusion-based Decentralized Federated Multi-Task Representation Learning
• Beyond Centralization: Provable Communication Efficient Decentralized Multi-Task Learning
• Bridging Lifelong and Multi-Task Representation Learning via Algorithm and Complexity Measure
• Exploiting Shared Representations for Personalized Federated Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• An Information-Theoretic Analysis of Out-of-Distribution Generalization in Meta-Learning with Applications to Meta-RL
• Transformers are almost optimal metalearners for linear classification
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Robust Meta-learning for Mixed Linear Regression with Small Batches
• Transfer Learning in Regression with Influential Points
• Meta-Learning Linear Models for Molecular Property Prediction
• Robust and Adaptive Spectral Method for Representation Multi-Task Learning with Contamination
• How Important is the Train-Validation Split in Meta-Learning?
• Impact of Representation Learning in Linear Bandits
• Meta-Learning Bandit Policies by Gradient Ascent
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Related

• Conditional Meta-Learning of Linear Representations
• Meta-Learning and representation learner: A short theoretical note
• Meta-learning of shared linear representations beyond well-specified linear regression
• Meta-learning for mixed linear regression
• Nonlinear Meta-Learning Can Guarantee Faster Rates