On the Similarities of Embeddings in Contrastive Learning

2025/06/11 by Chungpa Lee, Lee, Chungpa, Lim, Sehee +4 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face and Expression Recognition #Face recognition and analysis #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2506.09781

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Contrastive learning operates on a simple yet effective principle: Embeddings of positive pairs are pulled together, while those of negative pairs are pushed apart. In this paper, we propose a unified framework for understanding contrastive learning through the lens of cosine similarity, and present two key theoretical insights derived from this framework. First, in full-batch settings, we show that perfect alignment of positive pairs is unattainable when negative-pair similarities fall below a threshold, and this misalignment can be mitigated by incorporating within-view negative pairs into the objective. Second, in mini-batch settings, smaller batch sizes induce stronger separation among negative pairs in the embedding space, i.e., higher variance in their similarities, which in turn degrades the quality of learned representations compared to full-batch settings. To address this, we propose an auxiliary loss that reduces the variance of negative-pair similarities in mini-batch settings. Empirical results show that incorporating the proposed loss improves performance in small-batch settings.

Citations

• Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses
• Mini-Batch Optimization of Contrastive Loss
• Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis
• Sigmoid Loss for Language Image Pre-Training
• Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance
• Combined Scaling for Zero-shot Transfer Learning
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision
• Decoupled Contrastive Learning
• Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning
• Self-Supervised Learning with Kernel Dependence Maximization
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Learning Transferable Visual Models From Natural Language Supervision
• Contrastive Learning Inverts the Data Generating Process
• Understanding the Behaviour of Contrastive Loss
• Exploring Simple Siamese Representation Learning
• Exploring Simple Siamese Representation Learning
• Spatiotemporal Contrastive Video Representation Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• What Makes for Good Views for Contrastive Learning?
• Supervised Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Contrastive Multiview Coding
• Contrastive Multiview Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Learning deep representations by mutual information estimation and maximization
• Representation Learning with Contrastive Predictive Coding
• Do Better ImageNet Models Transfer Better?
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Deep Residual Learning for Image Recognition

Cited by

• Autograder+: A Multi-Faceted AI Framework for Rich Pedagogical Feedback in Programming Education

Related

• A Survey on Data Curation for Visual Contrastive Learning: Why Crafting Effective Positive and Negative Pairs Matters
• Rethinking Positive Pairs in Contrastive Learning
• \mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs
• Investigating the Role of Negatives in Contrastive Representation Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning