Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learning

2025/09/02 by Yilang Zhang, Zhang, Yilang, Bingcong Li +3
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2509.02418

openalex publication_date 2025/09/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Utilizing task-invariant knowledge acquired from related tasks as prior information, meta-learning offers a principled approach to learning a new task with limited data records. Sample-efficient adaptation of this prior information is a major challenge facing meta-learning, and plays an important role because it facilitates training the sought task-specific model with just a few optimization steps. Past works deal with this challenge through preconditioning that speeds up convergence of the per-task training. Though effective in representing locally quadratic loss curvatures, simple linear preconditioning can be hardly potent with complex loss geometries. Instead of relying on a quadratic distance metric, the present contribution copes with complex loss metrics by learning a versatile distance-generating function, which induces a nonlinear mirror map to effectively capture and optimize a wide range of loss geometries. With suitable parameterization, this generating function is effected by an expressive neural network that is provably a valid distance. Analytical results establish convergence of not only the proposed method, but also all meta-learning approaches based on preconditioning. To attain gradient norm less than ε, the convergence rate of O(ε^-2) is on par with standard gradient-based meta-learning methods. Numerical tests on few-shot learning datasets demonstrate the superior empirical performance of the novel algorithm, as well as its rapid per-task convergence, which markedly reduces the number of adaptation steps, hence also accommodating large-scale meta-learning models.

Citations

• Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent
• Meta-Learning with a Geometry-Adaptive Preconditioner
• On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptation
• Meta Mirror Descent: Optimiser Learning for Fast Convergence
• Learning where to learn: Gradient sparsity in meta and continual\n learning
• Meta-learning the Learning Trends Shared Across Tasks
• Bilevel Optimization: Convergence Analysis and Enhanced Design
• A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Critic
• Meta-Learning with Network Pruning
• Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameters
• Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Meta-Learning with Implicit Gradients
• Meta-Learning with Warped Gradient Descent
• On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms
• A Closer Look at Few-shot Classification
• Meta-Learning with Differentiable Convex Optimization
• Meta-Curvature
• Meta-Learning for Low-Resource Neural Machine Translation
• Meta-Learning with Latent Embedding Optimization
• Bayesian Model-Agnostic Meta-Learning
• Meta-learning with differentiable closed-form solvers
• Meta-Learning for Semi-Supervised Few-Shot Classification
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Attention Is All You Need
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Low Data Drug Discovery with One-shot Learning
• Input Convex Neural Networks
• Matching Networks for One Shot Learning
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Adam: A Method for Stochastic Optimization
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks

Related

• Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent
• Meta-Learning via Learned Loss
• Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Meta-Learning with Warped Gradient Descent