Towards the Generalization of Contrastive Self-Supervised Learning

2021/11/01 by Weiran Huang, Huang, Weiran, Mingyang Yi +4 · 8 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Cancer-related molecular mechanisms research #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2111.00743

openalex publication_date 2021/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, self-supervised learning has attracted great attention, since it only requires unlabeled data for model training. Contrastive learning is one popular method for self-supervised learning and has achieved promising empirical performance. However, the theoretical understanding of its generalization ability is still limited. To this end, we define a kind of (σ,δ)-measure to mathematically quantify the data augmentation, and then provide an upper bound of the downstream classification error rate based on the measure. It reveals that the generalization ability of contrastive self-supervised learning is related to three key factors: alignment of positive samples, divergence of class centers, and concentration of augmented data. The first two factors are properties of learned representations, while the third one is determined by pre-defined data augmentation. We further investigate two canonical contrastive losses, InfoNCE and cross-correlation, to show how they provably achieve the first two factors. Moreover, we conduct experiments to study the third factor, and observe a strong correlation between downstream performance and the concentration of augmented data.

Citations

• Contrastive Multiview Coding
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• On Mutual Information Maximization for Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• What Makes for Good Views for Contrastive Learning?
• Momentum Contrast for Unsupervised Visual Representation Learning
• Predicting What You Already Know Helps: Provable Self-Supervised\n Learning
• Learning Multiple Layers of Features from Tiny Images
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• SimCSE: Simple Contrastive Learning of Sentence Embeddings

Cited by

• LIHE: Linguistic Instance-Split Hyperbolic-Euclidean Framework for Generalized Weakly-Supervised Referring Expression Comprehension
• Exploring Structural Degradation in Dense Representations for Self-supervised Learning
• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• Contrastive Self-Supervised Learning at the Edge: An Energy Perspective
• Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning
• CoUn: Empowering Machine Unlearning via Contrastive Learning
• Rate-Distortion Limits for Multimodal Retrieval: Theory, Optimal Codes, and Finite-Sample Guarantees
• Semantic Concentration for Self-Supervised Dense Representations Learning

Related

• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• An Augmentation-Aware Theory for Self-Supervised Contrastive Learning
• An Augmentation Overlap Theory of Contrastive Learning
• A Theoretical Study of Inductive Biases in Contrastive Learning
• Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning