Big Self-Supervised Models are Strong Semi-Supervised Learners

2020/06/17 by Ting Chen, Simon Kornblith, Chen, Ting +8 · 1 voice · 177 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #Multimodal Machine Learning Applications #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.10029

NeurIPS'2020. Code and pretrained models at https://github.com/google-research/simclr

arxiv published 2020/06/17 · arxiv created 2020/10/26 · arxiv updated 2020/10/27

Abstract

One paradigm for learning from few labeled examples while making best use of a large amount of unlabeled data is unsupervised pretraining followed by supervised fine-tuning. Although this paradigm uses unlabeled data in a task-agnostic way, in contrast to common approaches to semi-supervised learning for computer vision, we show that it is surprisingly effective for semi-supervised learning on ImageNet. A key ingredient of our approach is the use of big (deep and wide) networks during pretraining and fine-tuning. We find that, the fewer the labels, the more this approach (task-agnostic use of unlabeled data) benefits from a bigger network. After fine-tuning, the big network can be further improved and distilled into a much smaller one with little loss in classification accuracy by using the unlabeled examples for a second time, but in a task-specific way. The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2, supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge. This procedure achieves 73.9% ImageNet top-1 accuracy with just 1% of the labels (≤13 labeled images per class) using ResNet-50, a 10× improvement in label efficiency over the previous state-of-the-art. With 10% of labels, ResNet-50 trained with our method achieves 77.5% top-1 accuracy, outperforming standard supervised training with all of the labels.

Citations

• Semi-supervised Sequence Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning

Cited by

• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
• ModSSC: A Modular Framework for Semi-Supervised Classification on Heterogeneous Data
• Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective
• Learning Transferable Visual Models From Natural Language Supervision
• VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• Large-Scale Unsupervised Person Re-Identification with Contrastive Learning
• Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Coordinative Learning with Ordinal and Relational Priors for Volumetric Medical Image Segmentation
• Contrastive Learning Inverts the Data Generating Process
• Data-Efficient Reinforcement Learning with Self-Predictive\n Representations
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images
• SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning
• SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations
• Active Learning with Task-Driven Representations for Messy Pools
• Adversarially Adaptive Normalization for Single Domain Generalization
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Do Different Tracking Tasks Require Different Appearance Models?
• An Overview of Deep Semi-Supervised Learning
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• Improve Unsupervised Pretraining for Few-label Transfer
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Robust Pre-Training by Adversarial Contrastive Learning
• Vector-quantized Image Modeling with Improved VQGAN
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation
• Implicit Modeling for Transferability Estimation of Vision Foundation Models
• Exploring Structural Degradation in Dense Representations for Self-supervised Learning
• Mapping Hidden Heritage: Self-supervised Pre-training on High-Resolution LiDAR DEM Derivatives for Archaeological Stone Wall Detection
• Latent Diffusion Model without Variational Autoencoder
• Learning to Recognize Correctly Completed Procedure Steps in Egocentric Assembly Videos through Spatio-Temporal Modeling
• Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
• Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• Long-Tailed Recognition via Information-Preservable Two-Stage Learning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data
• Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
• Multiple Fusion Adaptation: A Strong Framework for Unsupervised Semantic Segmentation Adaptation
• GroupCoOp: Group-robust Fine-tuning via Group Prompt Learning
• Personalizing Pre-trained Models
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Enhancing Vehicle Detection under Adverse Weather Conditions with Contrastive Learning
• TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learning
• Efficient Self-supervised Vision Transformers for Representation Learning
• Hard Negative Mixing for Contrastive Learning
• Training Neural Networks with Fixed Sparse Masks
• DiSSECT: Structuring Transfer-Ready Medical Image Representations through Discrete Self-Supervision
• Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization
• Latent Zoning Network: A Unified Principle for Generative Modeling, Representation Learning, and Classification
• Revisiting ResNets: Improved Training and Scaling Strategies
• Contrastive Learning of Musical Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition
• Disentangling Content from Style to Overcome Shortcut Learning: A Hybrid Generative-Discriminative Learning Framework
• Dynamic Convolution for 3D Point Cloud Instance Segmentation
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data
• Video-Based MPAA Rating Prediction: An Attention-Driven Hybrid Architecture Using Contrastive Learning
• Florence: A New Foundation Model for Computer Vision
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback
• Self-Damaging Contrastive Learning
• Masked Contrastive Learning for Anomaly Detection
• Whitening for Self-Supervised Representation Learning
• Intriguing Properties of Contrastive Losses
• Clustering-based Feature Representation Learning for Oracle Bone Inscriptions Detection
• Multimodal Semi-Supervised Learning for 3D Objects
• CryptPEFT: Efficient and Private Neural Network Inference via Parameter-Efficient Fine-Tuning
• Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
• TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation
• Perch 2.0: The Bittern Lesson for Bioacoustics
• Benchmarking Foundation Models for Mitotic Figure Classification
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras
• Rep-MTL: Unleashing the Power of Representation-level Task Saliency for Multi-Task Learning
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• Improving Joint Embedding Predictive Architecture with Diffusion Noise
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Basic Reading Distillation
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
• LEAD: Exploring Logit Space Evolution for Model Selection
• Are Fewer Labels Possible for Few-shot Learning?
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Semi-Supervised Classification and Segmentation on High Resolution Aerial Images
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• BYOL works even without batch statistics
• Multi-Format Contrastive Learning of Audio Representations
• How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
• A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation
• Video Contrastive Learning with Global Context
• SEED: Self-supervised Distillation For Visual Representation
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• View Invariant Learning for Vision-Language Navigation in Continuous Environments
• Graph-MLP: Node Classification without Message Passing in Graph
• Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR
• Cross-Domain Sentiment Classification with In-Domain Contrastive Learning
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Self-Supervised Contrastive Learning for Multi-Label Images
• FixCLR: Negative-Class Contrastive Learning for Semi-Supervised Domain Generalization
• Resampling Augmentation for Time Series Contrastive Learning: Application to Remote Sensing
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• Self-supervised Learning for Semi-supervised Temporal Language Grounding
• Unsupervised Pre-training for Person Re-identification
• Semi-Supervised Text Classification via Self-Pretraining
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• Visual Pre-Training on Unlabeled Images using Reinforcement Learning
• Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology
• Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
• QSEA: Quantum Self-supervised Learning with Entanglement Augmentation
• Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
• Rethinking Semi-supervised Segmentation Beyond Accuracy: Reliability and Robustness
• MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models
• Differentially Private Learning Needs Better Features (or Much More Data)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• Contrastive Learning Improves Model Robustness Under Label Noise
• Multi-source Few-shot Domain Adaptation
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Meta-Learning to Improve Pre-Training
• Improving Transformation Invariance in Contrastive Representation Learning
• eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• Unleashing the Potential of Unsupervised Pre-Training with Intra-Identity Regularization for Person Re-Identification
• How PARTs assemble into wholes: Learning the relative composition of images
• Aligned Contrastive Loss for Long-Tailed Recognition
• Positional Contrastive Learning for Volumetric Medical Image Segmentation
• PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models
• SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learning
• Diversify and Conquer: Open-set Disagreement for Robust Semi-supervised Learning with Outliers
• Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems
• Multi-Task Self-Supervised Pre-Training for Music Classification
• Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation
• CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Learning Object-Centric Video Models by Contrasting Sets
• Exploring the Possibility of TypiClust for Low-Budget Federated Active Learning
• The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation
• Generative AI and foundation models in medical image
• Self-Organizing Visual Prototypes for Non-Parametric Representation Learning
• LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Investigating Fine- and Coarse-grained Structural Correspondences Between Deep Neural Networks and Human Object Image Similarity Judgments Using Unsupervised Alignment
• From Pixels to Images: A Structural Survey of Deep Learning Paradigms in Remote Sensing Image Semantic Segmentation
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Semi-Supervised Vision Transformers
• Language Models That Walk the Talk: A Framework for Formal Fairness Certificates
• Model alignment using inter-modal bridges
• H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos
• Hybrid Discriminative-Generative Training via Contrastive Learning
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability
• Predicting butterfly species presence from satellite imagery using soft contrastive regularisation
• Contrastive Active Inference
• An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Self-supervised Representation Learning with Relative Predictive Coding
• Meta Clustering Learning for Large-scale Unsupervised Person Re-identification
• Model Connectomes: A Generational Approach to Data-Efficient Language Models
• Weak-to-Strong Generalization via Direct On-Policy Distillation
• Faster Predictive Coding Networks via Better Initialization
• Inverse Problems Leveraging Pre-trained Contrastive Representations
• Estimating Galactic Distances From Images Using Self-supervised Representation Learning
• StackMix: A complementary Mix algorithm
• Towards Anomaly-Aware Pre-Training and Fine-Tuning for Graph Anomaly Detection
• Impact of Language Guidance: A Reproducibility Study
• Pose-Aware Weakly-Supervised Action Segmentation

Discussions

• Big Self-Supervised Models Are Strong Semi-Supervised Learners [hn, 2 points, 0 comments]

Related

• Billion-scale semi-supervised learning for image classification
• Self-supervised Pretraining of Visual Features in the Wild
• A Simple Recipe for Competitive Low-compute Self supervised Vision Models
• Semi-supervised Vision Transformers at Scale
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models