Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning

2025/06/04 by Achleshwar Luthra, Luthra, Achleshwar, Tianbao Yang +3
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.04411

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing that standard CL objectives implicitly approximate a supervised variant we call the negatives-only supervised contrastive loss (NSCL), which excludes same-class contrasts. We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under a bound that is both label-agnostic and architecture-independent. We characterize the geometric structure of the global minimizers of the NSCL loss: the learned representations exhibit augmentation collapse, within-class collapse, and class centers that form a simplex equiangular tight frame. We further introduce a new bound on the few-shot error of linear-probing. This bound depends on two measures of feature variability--within-class dispersion and variation along the line between class centers. We show that directional variation dominates the bound and that the within-class dispersion's effect diminishes as the number of labeled samples increases. These properties enable CL and NSCL-trained representations to support accurate few-shot label recovery using simple linear probes. Finally, we empirically validate our theoretical findings: the gap between CL and NSCL losses decays at a rate of O((1)/(#classes)); the two losses are highly correlated; minimizing the CL loss implicitly brings the NSCL loss close to the value achieved by direct minimization; and the proposed few-shot error bound provides a tight estimate of probing performance in practice. The code and project page of the paper are available at [\hrefhttps://github.com/DLFundamentals/understanding-sslcode, \hrefhttps://dlfundamentals.github.io/ssl-is-approximately-sl/project page].

Citations

• Projection Head is Secretly an Information Bottleneck
• Clustering Properties of Self-Supervised Learning
• A Theoretical Characterization of Optimal Data Augmentations in Self-Supervised Learning
• Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model
• Optimal Sample Complexity of Contrastive Learning
• Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss
• Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching
• Reverse Engineering Self-Supervised Learning
• InfoNCE Loss Provably Learns Cluster-Preserving Representations
• Generalization Bounds for Few-Shot Transfer Learning with Pretrained Classifiers
• A Theoretical Study of Inductive Biases in Contrastive Learning
• Are All Losses Created Equal: A Neural Collapse Perspective
• Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning
• Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods
• Do More Negative Samples Necessarily Hurt in Contrastive Learning?
• Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations
• Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap
• On the Optimization Landscape of Neural Collapse under MSE Loss: Global\n Optimality with Unconstrained Features
• Understanding Contrastive Learning Requires Incorporating Inductive Biases
• Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
• Extended Unconstrained Features Model for Exploring Deep Neural Collapse
• Understanding Deep Contrastive Learning via Coordinate-wise Optimization
• On the Role of Neural Collapse in Transfer Learning
• Deep ViT Features as Dense Visual Descriptors
• Masked Autoencoders Are Scalable Vision Learners
• TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
• Decoupled Contrastive Learning
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• On the Surrogate Gap between Contrastive and Supervised Losses
• Pairwise Supervised Contrastive Learning of Sentence Representations
• Investigating the Role of Negatives in Contrastive Representation Learning
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path
• Toward Understanding the Feature Learning Process of Self-supervised\n Contrastive Learning
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• VICReg: Variance-Invariance-Covariance Regularization for\n Self-Supervised Learning
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• An Empirical Study of Training Self-Supervised Vision Transformers
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Learning Transferable Visual Models From Natural Language Supervision
• Probing Classifiers: Promises, Shortcomings, and Advances
• Contrastive Learning Inverts the Data Generating Process
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Understanding the Behaviour of Contrastive Loss
• Neural collapse with unconstrained features
• Intriguing Properties of Contrastive Losses
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Contrastive learning, multi-view redundancy, and linear models
• Predicting What You Already Know Helps: Provable Self-Supervised\n Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Language Models are Few-Shot Learners
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• What Makes for Good Views for Contrastive Learning?
• Prototypical Contrastive Learning of Unsupervised Representations
• Improved Baselines with Momentum Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Self-Supervised Learning of Pretext-Invariant Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• On Mutual Information Maximization for Representation Learning
• Contrastive Multiview Coding
• Contrastive Multiview Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Similarity of Neural Network Representations Revisited
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Formal Limitations on the Measurement of Mutual Information
• Learning deep representations by mutual information estimation and maximization
• Representation Learning with Contrastive Predictive Coding
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Large Batch Training of Convolutional Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Matching Networks for One Shot Learning
• Deep Residual Learning for Image Recognition
• CERT: Contrastive Self-supervised Learning for Language Understanding

Related

• On the Alignment Between Supervised and Self-Supervised Contrastive Learning
• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• Towards the Generalization of Contrastive Self-Supervised Learning
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least