Contrastive learning, multi-view redundancy, and linear models

2020/08/24 by Christopher Tosh, Akshay Krishnamurthy, Tosh, Christopher +3 · 11 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Music and Audio Processing #Text and Document Classification Technologies #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2008.10150

openalex publication_date 2020/08/24 · arxiv created 2021/04/14 · arxiv updated 2021/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.

Citations

• Dirichlet draws are sparse with high probability
• Learning deep representations by mutual information estimation and maximization
• Provably efficient RL with Rich Observations via Latent State Decoding
• Contrastive Multiview Coding
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Contrastive estimation reveals topic posterior information to linear models
• Predicting What You Already Know Helps: Provable Self-Supervised Learning

Cited by

• SAVeD: Semantic Aware Version Discovery
• On the Alignment Between Supervised and Self-Supervised Contrastive Learning
• Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
• Disentanglement of Variations with Multimodal Generative Modeling
• IndiSeek learns information-guided disentangled representations
• The Complexity of Finding Local Optima in Contrastive Learning
• DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition
• Causal Representation Learning with Observational Grouping for CXR Classification
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
• Representation Learning via Non-Contrastive Mutual Information

Related

• Rethinking Minimal Sufficient Representation in Contrastive Learning
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Self-supervised Learning from a Multi-view Perspective
• Understanding Self-supervised Contrastive Learning through Supervised Objectives
• A Principled Framework for Multi-View Contrastive Learning