Semi-Supervised Contrastive Learning with Orthonormal Prototypes

2025/11/27 by Li, Huanran, Nguyen, Manh, Pimentel-Alarcón, Daniel
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2512.07880

openalex publication_date 2025/11/27 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first identify a critical learning-rate threshold, beyond which standard contrastive losses converge to collapsed solutions. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP improves performance in image classification and object detection tasks while also exhibiting greater stability across different learning rates and batch sizes.

Citations

• Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation
• Simple and Asymmetric Graph Contrastive Learning without Augmentations
• Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss
• SimMatchV2: Semi-Supervised Learning with Graph Consistency
• Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects
• Balanced Contrastive Learning for Long-Tailed Visual Recognition
• Rethinking Minimal Sufficient Representation in Contrastive Learning
• SimMatch: Semi-supervised Learning with Similarity Matching
• Class-Aware Contrastive Semi-Supervised Learning
• Supervised Contrastive Learning for Recommendation
• Targeted Supervised Contrastive Learning for Long-Tailed Recognition
• Decoupled Contrastive Learning
• Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation
• Parametric Contrastive Learning
• Semi-supervised Contrastive Learning with Similarity Co-calibration
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• VICReg: Variance-Invariance-Covariance Regularization for\n Self-Supervised Learning
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning
• Boosting Contrastive Self-Supervised Learning with False Negative Cancellation
• A Survey on Contrastive Self-supervised Learning
• What Should Not Be Contrastive in Contrastive Learning
• Semi-Supervised Contrastive Learning with Generalized Contrastive Loss and Its Application to Speaker Recognition
• What Makes for Good Views for Contrastive Learning?
• Prototypical Contrastive Learning of Unsupervised Representations
• Supervised Contrastive Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Contrastive Multiview Coding
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• AutoAugment: Learning Augmentation Policies from Data
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Deep Residual Learning for Image Recognition
• A Large-Scale Car Dataset for Fine-Grained Categorization and Verification
• Fine-Grained Visual Classification of Aircraft
• Deep Fusion: Capturing Dependencies in Contrastive Learning via Transformer Projection Heads

Related

• Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)
• Collapse-Proof Non-Contrastive Self-Supervised Learning
• CLAF: Contrastive Learning with Augmented Features for Imbalanced Semi-Supervised Learning
• Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse
• Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features