Prototypical Contrastive Learning of Unsupervised Representations

2020/05/11 by Junnan Li, Pan Zhou, Li, Junnan +5 · 63 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2005.04966

openalex publication_date 2020/05/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.

Citations

• The Pascal Visual Object Classes (VOC) Challenge
• Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Improved Baselines with Momentum Contrastive Learning
• Momentum Contrast for Unsupervised Visual Representation Learning

Cited by

• SCS-SupCon: Sigmoid-based Common and Style Supervised Contrastive Learning with Adaptive Decision Boundaries
• Semi-Supervised Contrastive Learning with Orthonormal Prototypes
• RLM: A Vision-Language Model Approach for Radar Scene Understanding
• You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering
• Unsupervised Image Classification with Adaptive Nearest Neighbor Selection and Cluster Ensembles
• Point Cloud Quantization through Multimodal Prompting for 3D Understanding
• SiamMM: A Mixture Model Perspective on Deep Unsupervised Learning
• Beyond Augmentation: Leveraging Inter-Instance Relation in Self-Supervised Representation Learning
• Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learning
• ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learning
• A Multi-dimensional Semantic Surprise Framework Based on Low-Entropy Semantic Manifolds for Fine-Grained Out-of-Distribution Detection
• Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• Unsupervised Dynamic Feature Selection for Robust Latent Spaces in Vision Tasks
• Personalized federated prototype learning in mixed heterogeneous data scenarios
• Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts
• NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding
• GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learning
• An Adaptor for Triggering Semi-Supervised Learning to Out-of-Box Serve Deep Image Clustering
• Understanding Self-supervised Learning with Dual Deep Networks
• Unsupervised Object-Level Representation Learning from Scene Images
• 応用物理学会のレター誌"Applied Physics Express(APEX)"について(学界ニュース)
• Multi-Label Image Classification with Contrastive Learning
• Self-supervised Pretraining of Visual Features in the Wild
• Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images
• Bootstrap your own latent: A new approach to self-supervised Learning
• Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
• Contrastive Learning with Stronger Augmentations
• Contrastive Predictive Coding for Anomaly Detection
• SL-SLR: Self-Supervised Representation Learning for Sign Language Recognition
• DUDE: Diffusion-Based Unsupervised Cross-Domain Image Retrieval
• Valid Property-Enhanced Contrastive Learning for Targeted Optimization & Resampling for Novel Drug Design
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning
• Intriguing Properties of Contrastive Losses
• SoundCLR: Contrastive Learning of Representations For Improved Environmental Sound Classification
• Center-Oriented Prototype Contrastive Clustering
• Supporting Clustering with Contrastive Learning
• A Generalized Learning Framework for Self-Supervised Contrastive Learning
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
• Selective Contrastive Learning for Weakly Supervised Affordance Grounding
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment
• Emerging Trends in Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation with Image-Level Supervision
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Self-Supervised Pretraining and Controlled Augmentation Improve Rare Wildlife Recognition in UAV Images
• LEAD: Exploring Logit Space Evolution for Model Selection
• Support-set bottlenecks for video-text representation learning
• SuperCM: Improving Semi-Supervised Learning and Domain Adaptation through differentiable clustering
• Emerging Properties in Self-Supervised Vision Transformers
• Cluster Contrast for Unsupervised Visual Representation Learning
• RCG: Safety-Critical Scenario Generation for Robust Autonomous Driving via Real-World Crash Grounding
• Divergence-Based Similarity Function for Multi-View Contrastive Learning
• Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion
• SEED: Self-supervised Distillation For Visual Representation
• Prototypical Contrast and Reverse Prediction: Unsupervised Skeleton Based Action Recognition
• Robust brain age estimation from structural MRI with contrastive learning
• Supercm: Revisiting Clustering for Semi-Supervised Learning

Related

• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Fine-Grained Representation Learning via Multi-Level Contrastive Learning without Class Priors
• A Simple Framework for Contrastive Learning of Visual Representations