SEED: Self-supervised Distillation For Visual Representation

2021/01/12 by Zhiyuan Fang, Jianfeng Wang, Fang, Zhiyuan +9 · 11 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2101.04731

Accepted as a conference paper at ICLR 2021

openalex publication_date 2021/01/12 · arxiv created 2021/04/15 · arxiv updated 2021/04/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper is concerned with self-supervised learning for small models. The problem is motivated by our empirical studies that while the widely used contrastive self-supervised learning method has shown great progress on large model training, it does not work well for small models. To address this problem, we propose a new learning paradigm, named SElf-SupErvised Distillation (SEED), where we leverage a larger network (as Teacher) to transfer its representational knowledge into a smaller architecture (as Student) in a self-supervised fashion. Instead of directly learning from unlabeled data, we train a student encoder to mimic the similarity score distribution inferred by a teacher over a set of instances. We show that SEED dramatically boosts the performance of small networks on downstream tasks. Compared with self-supervised baselines, SEED improves the top-1 accuracy from 42.2% to 67.6% on EfficientNet-B0 and from 36.3% to 68.2% on MobileNet-v3-Large on the ImageNet-1k dataset.

Citations

• FitNets: Hints for Thin Deep Nets
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Model compression via distillation and quantization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• LLM-Supported Manufacturing Mapping Generation
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Prototypical Contrastive Learning of Unsupervised Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Learning Multiple Layers of Features from Tiny Images

Cited by

• PSMamba: Progressive Self-supervised Vision Mamba for Plant Disease Recognition
• Super4DR: 4D Radar-centric Self-supervised Odometry and Gaussian-based Map Optimization
• When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
• Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
• Learning Task-Agnostic Representations through Multi-Teacher Distillation
• TTRV: Test-Time Reinforcement Learning for Vision Language Models
• HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training
• Simple Distillation Baselines for Improving Small Self-supervised Models
• PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization

Related

• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• A Simple Recipe for Competitive Low-compute Self supervised Vision Models
• DMT: Comprehensive Distillation with Multiple Self-supervised Teachers
• Distill on the Go: Online knowledge distillation in self-supervised learning
• Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation