Meta-learning with differentiable closed-form solvers

2018/05/21 by Luca Bertinetto, Bertinetto, Luca, João F. Henriques +5 · 16 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1805.08136

openalex publication_date 2018/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Few-Shot Learning with Graph Neural Networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• DUPLE: An Intelligent Cross-Deployment Recognition Framework for Fiber-Optic Perimeter Security under Scarce Target Labels
• Free Lunch to Meet the Gap: Intermediate Domain Reconstruction for Cross-Domain Few-Shot Learning
• Genomic Next-Token Predictors are In-Context Learners
• Solving bilevel optimization via sequential minimax optimization
• Sparse Model Inversion: Efficient Inversion of Vision Transformers for Data-Free Applications
• MetaCaDI: A Meta-Learning Framework for Scalable Causal Discovery with Unknown Interventions
• Bi-Level Optimization for Generative Recommendation: Bridging Tokenization and Generation
• SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse
• Provably Robust Adaptation for Language-Empowered Foundation Models
• Task-Level Contrastiveness for Cross-Domain Few-Shot Learning
• BATR-FST: Bi-Level Adaptive Token Refinement for Few-Shot Transformers
• Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learning
• Optimizing In-Context Learning for Efficient Full Conformal Prediction
• Proximal Mapping for Deep Regularization
• Self-Denoising Neural Networks for Few Shot Learning
• Slot Attention-based Feature Filtering for Few-Shot Learning

Related

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Deep Transfer Learning with Ridge Regression
• Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels
• Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning
• Cross-Domain Few-Shot Learning with Meta Fine-Tuning