Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables

2025/05/18 by Yu Gui, Cong Ma, Gui, Yu +3 · 4 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Statistics Theory (math.ST) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.12473

openalex publication_date 2025/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citepradford2021learning. In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.

Citations

• A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics
• A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
• Multimodal data integration and cross-modal querying via orchestrated approximate message passing
• Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
• A Theory of Multimodal Learning
• Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage
• A Cookbook of Self-Supervised Learning
• On the Importance of Contrastive Loss in Multimodal Learning
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
• Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data
• Multimodal Learning with Transformers: A Survey
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Attention Bottlenecks for Multimodal Fusion
• Self-Supervised Learning with Kernel Dependence Maximization
• What Makes Multi-modal Learning Better than Single (Provably)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Learning Transferable Visual Models From Natural Language Supervision
• Contrastive Learning Inverts the Data Generating Process
• Understanding the Behaviour of Contrastive Loss
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Learning Robust Representations via Multi-View Information Bottleneck
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• On Mutual Information Maximization for Representation Learning
• Contrastive Multiview Coding
• Contrastive Multiview Coding
• Learning Representations by Maximizing Mutual Information Across Views
• On Variational Bounds of Mutual Information
• Do ImageNet Classifiers Generalize to ImageNet?
• Learning deep representations by mutual information estimation and maximization
• Representation Learning with Contrastive Predictive Coding
• Multimodal Machine Learning: A Survey and Taxonomy
• Multimodal Machine Learning: A Survey and Taxonomy
• Jointly Modeling Embedding and Translation to Bridge Video and Language
• Deep Learning and the Information Bottleneck Principle
• Deep Multimodal Learning for Audio-Visual Speech Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Multimodal Deep Learning
• Multimodal fusion for multimedia analysis: a survey
• A kernel method for canonical correlation analysis
• Estimation of Entropy and Mutual Information
• Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning

Cited by

• IndiSeek learns information-guided disentangled representations
• Global Minimizers of Sigmoid Contrastive Loss
• Learning Molecular Representations from Cellular Phenotypes with Structure Preservation
• DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

Related

• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• Contrastive learning, multi-view redundancy, and linear models
• Transferability of Representations Learned using Supervised Contrastive Learning Trained on a Multi-Domain Dataset
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning
• Factorized Contrastive Learning: Going Beyond Multi-view Redundancy