Self-training for Few-shot Transfer Across Extreme Task Differences

2020/10/15 by Cheng Perng Phoo, Bharath Hariharan, Phoo, Cheng Perng +1 · 4 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2010.07734

openalex publication_date 2020/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most few-shot learning techniques are pre-trained on a large, labeled "base dataset". In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different "source" problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https://github.com/cpphoo/STARTUP.

Citations

• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Conditional Adversarial Domain Adaptation
• Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
• Cross Attention Network for Few-shot Classification
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
• Self-Training With Noisy Student Improves ImageNet Classification
• Momentum Contrast for Unsupervised Visual Representation Learning

Cited by

• Free Lunch to Meet the Gap: Intermediate Domain Reconstruction for Cross-Domain Few-Shot Learning
• FreqGRL: Suppressing Low-Frequency Bias and Mining High-Frequency Knowledge for Cross-Domain Few-Shot Learning
• Enhancing Pseudo-Boxes via Data-Level LiDAR-Camera Fusion for Unsupervised 3D Object Detection
• Beyond Class Tokens: LLM-guided Dominant Property Mining for Few-shot Classification

Related

• Cross-domain few-shot learning with unlabelled data
• Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty
• Self-Supervised Prototypical Transfer Learning for Few-Shot Classification
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• A Cross-Domain Few-Shot Learning Method Based on Domain Knowledge Mapping