A Survey on Contrastive Self-supervised Learning

2020/10/31 by Ashish Jaiswal, Ashwin Ramesh Babu, Jaiswal, Ashish +7 · 65 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.2011.00362

20 pages, 18 figures, 6 tables

arxiv created 2021/02/07 · arxiv updated 2021/02/09

Abstract

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudo labels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning methods for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we have a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make substantial progress.

Cited by

• Self-Supervised Consistency Enhanced Disentangled Learning for Neural Decoding Generalization in Brain-Machine Interface
• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
• Recurrent Video Masked Autoencoders
• Supervised Contrastive Frame Aggregation for Video Representation Learning
• Recover-to-Forget: Gradient Reconstruction from LoRA for Efficient LLM Unlearning
• MANTA: Physics-Informed Generalized Underwater Object Tracking
• Semi-Supervised Contrastive Learning with Orthonormal Prototypes
• History-Augmented Contrastive Learning With Soft Mixture of Experts for Blind Super-Resolution of Planetary Remote Sensing Images
• Self-supervised learning for multi-label sewer defect classification
• Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions
• Contrastive Predictive Coding Done Right for Mutual Information Estimation
• Optimizing Retrieval for RAG via Reinforced Contrastive Learning
• HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series
• A geometric and deep learning reproducible pipeline for monitoring floating anthropogenic debris in urban rivers using in situ cameras
• CLEANet: Robust and Efficient Anomaly Detection in Contaminated Multivariate Time Series
• HIDISC: A Hyperbolic Framework for Domain Generalization with Generalized Category Discovery
• Towards Robust Artificial Intelligence: Self-Supervised Learning Approach for Out-of-Distribution Detection
• How much speech data is necessary for ASR in African languages? An evaluation of data scaling in Kinyarwanda and Kikuyu
• SSL-SE-EEG: A Framework for Robust Learning from Unlabeled EEG Data with Self-Supervised Learning and Squeeze-Excitation Networks
• SongFormer: Scaling Music Structure Analysis with Heterogeneous Supervision
• Contrastive Diffusion Guidance for Spatial Inverse Problems
• Experience Paper: Adopting Activity Recognition in On-demand Food Delivery Business
• Enhancing hyperspectral image prediction with contrastive learning in low-label regimes
• An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection
• Mask What Matters: Controllable Text-Guided Masking for Self-Supervised Medical Image Analysis
• Manifold-Aware Diffusion-Augmented Contrastive Learning for Noise-Robust Biosignal Representation
• CoUn: Empowering Machine Unlearning via Contrastive Learning
• Learning Representations in Video Game Agents with Supervised Contrastive Imitation Learning
• Investigating Location-Regularised Self-Supervised Feature Learning for Seafloor Visual Imagery
• Ontology-Aligned Embeddings for Data-Driven Labour Market Analytics
• Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learning
• A Generalized Learning Framework for Self-Supervised Contrastive Learning
• Contrastive Representations for Temporal Reasoning
• rETF-semiSL: Semi-Supervised Learning for Neural Collapse in Temporal Data
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• Breaking the Top-K Barrier: Advancing Top-K Ranking Metrics Optimization in Recommender Systems
• An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges
• CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data
• Generic Speech Enhancement with Self-Supervised Representation Space Loss
• When Graph Contrastive Learning Backfires: Spectral Vulnerability and Defense in Recommendation
• Clustering via Self-Supervised Diffusion
• Adaptability of ASR Models on Low-Resource Language: A Comparative Study of Whisper and Wav2Vec-BERT on Bangla
• The Hidden Link Between RLHF and Contrastive Learning
• AquaCluster: Using Satellite Images And Self-supervised Machine Learning Networks To Detect Water Hidden Under Vegetation
• CL-ISR: A Contrastive Learning and Implicit Stance Reasoning Framework for Misleading Text Detection on Social Media
• Advancing Symbolic Discovery on Unsupervised Data: A Pre-training Framework for Non-degenerate Implicit Equation Discovery
• A Tutorial-cum-Survey on Self-Supervised Learning for Wi-Fi Sensing: Trends, Challenges, and Outlook
• FreRA: A Frequency-Refined Augmentation for Contrastive Learning on Time Series Classification
• Subgraph Gaussian Embedding Contrast for Self-Supervised Graph Representation Learning
• FCKT: Fine-Grained Cross-Task Knowledge Transfer with Semantic Contrastive Learning for Targeted Sentiment Analysis
• Enhancing Contrastive Learning-based Electrocardiogram Pretrained Model with Patient Memory Queue
• Smartwatch Photoplethysmography-Derived Heart Age via ECG-Guided Cross-Modal Pretraining as a Digital Biomarker of Vascular Aging
• Rethinking Contrastive Learning in Graph Anomaly Detection: A Clean-View Perspective
• Contrastive Learning-Enhanced Trajectory Matching for Small-Scale Dataset Distillation
• Language Models That Walk the Talk: A Framework for Formal Fairness Certificates
• Towards Sustainability in 6G Network Slicing with Energy-Saving and Optimization Methods
• Joint Low-level and High-level Textual Representation Learning with Multiple Masking Strategies
• SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Images
• Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines
• Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects
• Automated Measurement of Eczema Severity with Self-Supervised Learning
• MusFlow: Multimodal Music Generation via Conditional Flow Matching
• A survey on recent trends in robotics and artificial intelligence in the furniture industry
• On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction
• JEPA4Rec: Learning Effective Language Representations for Sequential Recommendation via Joint Embedding Predictive Architecture
• Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generation
• On the Importance of Conditioning for Privacy-Preserving Data Augmentation

Related

• Brief Introduction to Contrastive Learning Pretext Tasks for Visual Representation
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Self-supervised Learning: Generative or Contrastive
• A Survey on Self-supervised Contrastive Learning for Multimodal Text-Image Analysis
• Contrastive Learning with Adversarial Examples