SLIP: Self-supervision meets Language-Image Pre-training

2021/12/23 by Norman Mu, Alexander M. Kirillov, Mu, Norman +5 · 34 citations
Computer Science · Medicine · #COVID-19 diagnosis using AI #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2112.12750

openalex publication_date 2021/12/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across ImageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

Cited by

• CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
• ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching
• SuperCLIP: CLIP with Simple Classification Supervision
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP
• Learning complete and explainable visual representations from itemized text supervision
• Online Segment Any 3D Thing as Instance Tracking
• PowerCLIP: Powerset Alignment for Contrastive Pre-Training
• Rethinking Cross-Generator Image Forgery Detection through DINOv3
• RLM: A Vision-Language Model Approach for Radar Scene Understanding
• ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis
• PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning
• LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction
• Point Cloud Quantization through Multimodal Prompting for 3D Understanding
• MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition
• Model-Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn't the Right One
• Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
• CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder
• Leveraging Shared Prototypes for a Multimodal Pulse Motion Foundation Model
• TextCAM: Explaining Class Activation Map with Text
• SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP
• GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learning
• InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions
• Self-Supervised Cross-Modal Learning for Image-to-Point Cloud Registration
• Multimodal Representation Learning Conditioned on Semantic Relations
• GLAM: Geometry-Guided Local Alignment for Multi-View VLP in Mammography
• MobileCLIP2: Improving Multi-Modal Reinforced Training
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine
• MobileViCLIP: An Efficient Video-Text Model for Mobile Devices
• CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment
• HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees
• MSGCoOp: Multiple Semantic-Guided Context Optimization for Few-Shot Learning
• Meta CLIP 2: A Worldwide Scaling Recipe

Related

• Image Re-Identification: Where Self-supervision Meets Vision-Language Learning
• Harmony: A Joint Self-Supervised and Weakly-Supervised Framework for Learning General Purpose Visual Representations
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• SiT: Self-supervised vIsion Transformer