Whitening for Self-Supervised Representation Learning

2020/07/13 by Ermolov, Aleksandr, Aliaksandr Siarohin, Enver Sangineto +4 · 13 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2007.06346

openalex publication_date 2020/07/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most of the current self-supervised representation learning (SSL) methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance ("positives") are contrasted with instances extracted from other images ("negatives"). For the learning to be effective, many negatives should be compared with a positive pair, which is computationally demanding. In this paper, we propose a different direction and a new loss function for SSL, which is based on the whitening of the latent-space features. The whitening operation has a "scattering" effect on the batch samples, avoiding degenerate solutions where all the sample representations collapse to a single point. Our solution does not require asymmetric networks and it is conceptually simple. Moreover, since negatives are not needed, we can extract multiple positive pairs from the same image instance. The source code of the method and of all the experiments is available at: https://github.com/htdt/self-supervised.

Citations

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• In Defense of the Triplet Loss for Person Re-Identification
• Learning deep representations by mutual information estimation and maximization
• Whitening and Coloring batch transform for GANs
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• On Mutual Information Maximization for Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Bootstrap your own latent: A new approach to self-supervised Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• Contrastive Multiview Coding
• Learning Multiple Layers of Features from Tiny Images

Cited by

• APT: Affine Prototype-Timestamp For Time Series Forecasting Under Distribution Shift
• Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines
• CoMA: Complementary Masking and Hierarchical Dynamic Multi-Window Self-Attention in a Unified Pre-training Framework
• A Theory of Contrastive Learning with Natural Images
• T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning
• Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density
• Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes
• Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
• Self-Supervised Representation Learning as Mutual Information Maximization
• GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learning
• An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection
• A Generalized Learning Framework for Self-Supervised Contrastive Learning
• Foundation Model for Skeleton-Based Human Action Understanding

Related

• Whitening Consistently Improves Self-Supervised Learning
• An Investigation into Whitening Loss for Self-supervised Learning
• Direct Coloring for Self-Supervised Enhanced Feature Decoupling
• Modulate Your Spectrum in Self-Supervised Learning
• Trip-ROMA: Self-Supervised Learning with Triplets and Random Mappings