Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets

2020/06/25 by Haoxiang Wang, Ruoyu Sun, Wang, Haoxiang +3
Computer Science · Engineering · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2006.14606

openalex publication_date 2020/06/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Gradient-based meta-learning (GBML) with deep neural nets (DNNs) has become a popular approach for few-shot learning. However, due to the non-convexity of DNNs and the bi-level optimization in GBML, the theoretical properties of GBML with DNNs remain largely unknown. In this paper, we first aim to answer the following question: Does GBML with DNNs have global convergence guarantees? We provide a positive answer to this question by proving that GBML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate. The second question we aim to address is: How does GBML achieve fast adaption to new tasks with prior experience on past tasks? To answer it, we theoretically show that GBML is equivalent to a functional gradient descent operation that explicitly propagates experience from the past tasks to new ones, and then we prove a generalization error bound of GBML with over-parameterized DNNs.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Domain Generalization via Invariant Feature Representation
• On First-Order Meta-Learning Algorithms
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• A Convergence Theory for Deep Learning via Over-Parameterization
• Elimination of All Bad Local Minima in Deep Learning
• Online Meta-Learning
• Meta-Learning with Implicit Gradients
• A Sample Complexity Separation between Non-Convex and Convex\n Meta-Learning
• Generalizing from a Few Examples
• Generalizing to unseen domains via distribution matching

Related

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Towards Understanding Generalization in Gradient-Based Meta-Learning
• Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning
• Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation
• Gradient Agreement as an Optimization Objective for Meta-Learning