Barlow Twins: Self-Supervised Learning via Redundancy Reduction
2021/03/04 by Jure Žbontar, Jure Zbontar, Jing Li +9 · 260 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Biological sciences #FOS: Computer and information sciences #Image Processing Techniques and Applications #Machine Learning (cs.LG) #Neurons and Cognition (q-bio.NC) #cs.AI #cs.CV #cs.LG #q-bio.NC
paper · pdf · doi:10.48550/arxiv.2103.03230
13 pages, 6 figures, to appear at ICML 2021
openalex publication_date 2021/03/04 · arxiv created 2021/06/14 · arxiv updated 2021/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.
Citations
Cited by
- Mosaic: A Fleet of User Embedding Specialists for Recommendation at Meta
- SPECTRE: Spectral Pre-training Embeddings with Cylindrical Temporal Rotary Position Encoding for Fine-Grained sEMG-Based Movement Decoding
- Multi-Head Spectral-Adaptive Graph Anomaly Detection
- The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
- Robustifying pathology foundation models via fine-tuning
- Disentangling Multi-View Scanning in Mamba for Network Traffic Anomaly Detection
- Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines
- Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
- KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning
- brat: Aligned Multi-View Embeddings for Brain MRI Analysis
- Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling
- SCS-SupCon: Sigmoid-based Common and Style Supervised Contrastive Learning with Adaptive Decision Boundaries
- Magnification-Aware Distillation (MAD): A Self-Supervised Framework for Unified Representation Learning in Gigapixel Whole-Slide Images
- From Feature Interaction to Feature Generation: A Generative Paradigm of CTR Prediction Models
- Supervised Contrastive Frame Aggregation for Video Representation Learning
- Transferring Clinical Knowledge into ECGs Representation
- SARL: Spatially-Aware Self-Supervised Representation Learning for Visuo-Tactile Perception
- Semi-Supervised Contrastive Learning with Orthonormal Prototypes
- Pre-train to Gain: Robust Learning Without Clean Labels
- Learning Scalable Temporal Representations in Spiking Neural Networks Without Labels
- In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithm
- TimePre: Bridging Accuracy, Efficiency, and Stability in Probabilistic Time-Series Forecasting
- Self-Supervised Learning by Curvature Alignment
- APT: Affine Prototype-Timestamp For Time Series Forecasting Under Distribution Shift
- Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines
- PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
- DI3CL: Contrastive Learning With Dynamic Instances and Contour Consistency for SAR Land-Cover Classification Foundation Model
- FlowFeat: Pixel-Dense Embedding of Motion Profiles
- Adaptive Multi-view Graph Contrastive Learning via Fractional-order Neural Diffusion Networks
- SiamMM: A Mixture Model Perspective on Deep Unsupervised Learning
- An Augmentation Overlap Theory of Contrastive Learning
- Barlow Twins for Sequential Recommendation
- Controlling Contrastive Self-Supervised Learning with Knowledge-Driven Multiple Hypothesis: Application to Beat Tracking
- Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification
- A Theory of Contrastive Learning with Natural Images
- Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography
- Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method
- Eigenfunction Extraction for Ordered Representation Learning
- T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning
- Model-Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn't the Right One
- Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projections
- Beyond Augmentation: Leveraging Inter-Instance Relation in Self-Supervised Representation Learning
- AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing
- Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions
- Interpretable Multimodal Zero-Shot ECG Diagnosis via Structured Clinical Knowledge Alignment
- SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse
- Neural Diversity Regularizes Hallucinations in Language Models
- A theoretical framework for self-supervised contrastive learning for continuous dependent data
- Towards the Generalization of Contrastive Self-Supervised Learning
- CytoNet: A Foundation Model for the Human Cerebral Cortex at Cellular Resolution
- Exploring Structural Degradation in Dense Representations for Self-supervised Learning
- Vision Pair Learning: An Efficient Training Framework for Image Classification
- A Multimodal Approach to Heritage Preservation in the Context of Climate Change
- On the Optimal Representation Efficiency of Barlow Twins: An Information-Geometric Interpretation
- A compressed code for memory discrimination
- Understanding Self-supervised Contrastive Learning through Supervised Objectives
- Probabilistic Variational Contrastive Learning
- TIT: A Tree-Structured Instruction Tuning Approach for LLM-Based Code Translation
- Enhancing Self-Supervised Learning with Semantic Pairs A New Dataset and Empirical Study
- DM1: MeanFlow with Dispersive Regularization for 1-Step Robotic Manipulation
- A Systematic Evaluation of Self-Supervised Learning for Label-Efficient Sleep Staging with Wearable EEG
- Contrastive Self-Supervised Learning at the Edge: An Energy Perspective
- On the Alignment Between Supervised and Self-Supervised Contrastive Learning
- Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
- Conditional Representation Learning for Customized Tasks
- Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs
- Adapting HFMCA to Graph Data: Self-Supervised Learning for Generalizable fMRI Representations
- Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
- Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
- Self-Supervised Representation Learning as Mutual Information Maximization
- Uncovering the Computational Ingredients of Human-Like Representations in LLMs
- CODED-SMOOTHING: Coding Theory Helps Generalization
- Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
- BERT Bi-modal self-supervised learning for crop classification using Sentinel-2 and Planetscope
- Interpretable Self-Supervised Learning via Representer Landmarks and Nyström Approximation
- GenView++: Unifying Adaptive Generative Augmentation and Quality-Driven Supervision for Contrastive Representation Learning
- Disentanglement of Variations with Multimodal Generative Modeling
- An Investigation into the Performance of Non-Contrastive Self-Supervised Learning Methods for Network Intrusion Detection
- ZeroSiam: An Efficient Siamese for Test-Time Entropy Optimization without Collapse
- CLAD-Net: Continual Activity Recognition in Multi-Sensor Wearable Systems
- Can Local Learning Match Self-Supervised Backpropagation?
- MonoCon: A general framework for learning ultra-compact high-fidelity representations using monotonicity constraints
- A Data-driven Typology of Vision Models from Integrated Representational Metrics
- Discovering alternative solutions beyond the simplicity bias in recurrent neural networks
- Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation
- SiamJEPA: On the Role of Siamese Student Encoders in JEPA
- It's Not Just More Demos: Counterfactual Action Sensitivity Coverage for Data-Efficient Robust Robot Imitation
- Self-Supervised Learning with Kernel Dependence Maximization
- Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization
- Enhancing Semantic Segmentation with Continual Self-Supervised Pre-training
- Unlocking Hidden Potential in Point Cloud Networks with Attention-Guided Grouping-Feature Coordination
- CoUn: Empowering Machine Unlearning via Contrastive Learning
- DinoTwins: Combining DINO and Barlow Twins for Robust, Label-Efficient Vision Transformers
- Why all roads don't lead to Rome: Representation geometry varies across the human visual cortical hierarchy
- Barlow Graph Auto-Encoder for Unsupervised Network Embedding
- LayerLock: Non-collapsing Representation Learning with Progressive Freezing
- Semantic Concentration for Self-Supervised Dense Representations Learning
- Unsupervised Multi-Attention Meta Transformer for Rotating Machinery Fault Diagnosis
- Enhancing 3D Medical Image Understanding with Pretraining Aided by 2D Multimodal Large Language Models
- Self-Supervised Training Enhances Online Continual Learning
- Kernel VICReg for Self-Supervised Learning in Reproducing Kernel Hilbert Space
- Barlow-Swin: Toward a novel siamese-based segmentation architecture using Swin-Transformers
- Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairs
- Rethinking Supervised Pre-training for Better Downstream Transferring
- Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization
- Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
- SC-GIR: Goal-oriented Semantic Communication via Invariant Representation Learning
- Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations
- A Model-agnostic Strategy to Mitigate Embedding Degradation in Personalized Federated Recommendation
- A Generalized Learning Framework for Self-Supervised Contrastive Learning
- EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition
- Foundation Model for Skeleton-Based Human Action Understanding
- NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
- Learning State-Space Models of Dynamic Systems from Arbitrary Data using Joint Embedding Predictive Architectures
- VIFSS: View-Invariant and Figure Skating-Specific Pose Representation Learning for Temporal Action Segmentation
- PatchGame: Learning to Signal Mid-level Patches in Referential Games
- MCLPD:Multi-view Contrastive Learning for EEG-based PD Detection Across Datasets
- When Is Prior Knowledge Helpful? Exploring the Evaluation and Selection of Unsupervised Pretext Tasks from a Neuro-Symbolic Perspective
- Learning Representations for Pixel-based Control: What Matters and Why?
- CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment
- Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention
- CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework
- BEVCon: Advancing Bird's Eye View Perception with Contrastive Learning
- Bypassing Skip-Gram Negative Sampling: Dimension Regularization as a More Efficient Alternative for Graph Embeddings
- PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective
- Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks
- BarlowWalk: Self-supervised Representation Learning for Legged Robot Terrain-adaptive Locomotion
- Balancing Information Preservation and Disentanglement in Self-Supervised Music Representation Learning
- Cross-Architecture Distillation Made Simple with Redundancy Suppression
- Foundation Models and Transformers for Anomaly Detection: A Survey
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation
- A Contrastive Diffusion-based Network (CDNet) for Time Series Classification
- Self-Supervised Neural Architecture Search for Imbalanced Datasets
- MatSSL: Robust Self-Supervised Representation Learning for Metallographic Image Segmentation
- RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks
- DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
- Leveraging unlabeled SEM datasets with self-supervised learning for enhanced particle segmentation
- Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
- A Study on Variants of Conventional, Fuzzy, and Nullspace-Based Independence Criteria for Improving Supervised and Unsupervised Learning
- Emerging Properties in Self-Supervised Vision Transformers
- Cluster Contrast for Unsupervised Visual Representation Learning
- From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
- Diffuse and Disperse: Image Generation with Representation Regularization
- CLA: Latent Alignment for Online Continual Self-Supervised Learning
- Robust Contrastive Learning Using Negative Samples with Diminished Semantics
- Task Priors: Enhancing Model Evaluation by Considering the Entire Space of Downstream Tasks
- Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
- Objectomaly: Objectness-Aware Refinement for OoD Segmentation with Structural Consistency and Boundary Precision
- Zero-Shot Neural Architecture Search with Weighted Response Correlation
- Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images
- Deconfounding Causal Inference through Two-Branch Framework with Early-Forking for Sensor-Based Cross-Domain Activity Recognition
- Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
- Dual-Alignment Knowledge Retention for Continual Medical Image Segmentation
- A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
- Subject Invariant Contrastive Learning for Human Activity Recognition
- From Video to EEG: Adapting Joint Embedding Predictive Architecture to Uncover Saptiotemporal Dynamics in Brain Signal Analysis
- Large-Scale Hyperspectral Image Clustering Using Contrastive Learning
- Wildlife Target Re-Identification Using Self-supervised Learning in Non-Urban Settings
- Robust brain age estimation from structural MRI with contrastive learning
- Compressive Visual Representations
- Aligning Pretraining for Detection via Object-Level Contrastive Learning
- Reducing Variability of Multiple Instance Learning Methods for Digital Pathology
- Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning
- TRIM: A Self-Supervised Video Summarization Framework Maximizing Temporal Relative Information and Representativeness
- InvZW: Invariant Feature Learning via Noise-Adversarial Training for Robust Image Zero-Watermarking
- TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis
- Multiple Object Stitching for Unsupervised Representation Learning
- Resampling Augmentation for Time Series Contrastive Learning: Application to Remote Sensing
- Leveraging neural network interatomic potentials for a foundation model of chemistry
- Variational Supervised Contrastive Learning
- DeInfoReg: A Decoupled Learning Framework for Better Training Throughput
- Enhancing VICReg: Random-Walk Pairing for Improved Generalization and Better Global Semantics Capturing
- A Survey of State Representation Learning for Deep Reinforcement Learning
- Self-supervised Feature Extraction for Enhanced Ball Detection on Soccer Robots
- Bridging Brain with Foundation Models through Self-Supervised Learning
- Cluster Analysis with Deep Embeddings and Contrastive Learning
- Evaluating the fairness of fine-tuning strategies in self-supervised learning
- Dual Perspectives on Non-Contrastive Self-Supervised Learning
- Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments
- Contrastive Self-Supervised Learning As Neural Manifold Packing
- Advancing Image-Based Grapevine Variety Classification with a New Benchmark and Evaluation of Masked Autoencoders
- Towards Demystifying Representation Learning with Non-contrastive Self-supervision
- Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reports
- Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology
- EquiCaps: Predictor-Free Pose-Aware Pre-Trained Capsule Networks
- Deep Neural Compression Via Concurrent Pruning and Self-Distillation
- Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
- How PARTs assemble into wholes: Learning the relative composition of images
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
- seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
- Simple, Good, Fast: Self-Supervised World Models Free of Baggage
- Self-Supervised Multi-View Representation Learning using Vision-Language Model for 3D/4D Facial Expression Recognition
- Accurate Estimation of Mutual Information in High Dimensional Data
- A Mathematical Perspective On Contrastive Learning
- GARLIC: GAussian Representation LearnIng for spaCe partitioning
- A Tutorial-cum-Survey on Self-Supervised Learning for Wi-Fi Sensing: Trends, Challenges, and Outlook
- Self-supervised feature learning for cardiac Cine MR image reconstruction
- "Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shops
- Spatio-Temporal Joint Density Driven Learning for Skeleton-Based Action Recognition
- Weakly-Supervised Contrastive Learning for Imprecise Class Labels
- An Augmentation-Aware Theory for Self-Supervised Contrastive Learning
- Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems
- DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces
- Enhancing CTR Prediction with De-correlated Expert Networks
- Imagine Beyond! Distributionally Robust Auto-Encoding for State Space Coverage in Online Reinforcement Learning
- Self-Supervised Representations for Binary Program Clustering: From Empirical Study to Retrieval-Augmented Learning
- Stochastic Forward-Forward Learning through Representational Dimensionality Compression
- Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning
- An Empirical Study of Graph Contrastive Learning
- An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
- Learning From Long-Tailed Data With Noisy Labels
- Generalized Category Discovery via Token Manifold Capacity Learning
- RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data
- Pushing the Frontiers of Self-Distillation Prototypes Network with Dimension Regularization and Score Normalization
- Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
- PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization
- scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data
- scSiameseClu: A Siamese Clustering Framework for Interpreting single-cell RNA Sequencing Data
- AdaDim: Dimensionality Adaptation for SSL Representational Dynamics
- Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self Supervised Learning
- Ditch the Denoiser: Emergence of Noise Robustness in Self-Supervised Learning from Data Curriculum
- Fine-grained Contrastive Learning for ECG-Report Alignment with Waveform Enhancement
- Equally Critical: Samples, Targets, and Their Mappings in Datasets
- Natural Attribute-based Shift Detection
- Imputation-free and Alignment-free: Incomplete Multi-view Clustering Driven by Consensus Semantic Learning
- Domain-Agnostic Clustering with Self-Distillation
- MrTrack: Register Mamba for Needle Tracking with Rapid Reciprocating Motion during Ultrasound-Guided Aspiration Biopsy
- Unsupervised Multiview Contrastive Language-Image Joint Learning with Pseudo-Labeled Prompts Via Vision-Language Model for 3D/4D Facial Expression Recognition
- Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry
- Contextures: Representations from Contexts
- Enhancing User Sequence Modeling through Barlow Twins-based Self-Supervised Learning
- Self-Supervision Enhances Instance-based Multiple Instance Learning Methods in Digital Pathology: A Benchmark Study
- Test time Adaptation through Perturbation Robustness
- Self-Supervised Learning by Estimating Twin Class Distributions
- A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
- Next Embedding Prediction Makes World Models Stronger
- When Does LeJEPA Learn a World Model?
- Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning
- Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection
- Open-set Anomaly Segmentation in Complex Scenarios
- You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
- Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images
- Supervised Pretraining for Material Property Prediction
- ASMa: Asymmetric Spatio-temporal Masking for Skeleton Action Representation Learning
- A Genealogy of Foundation Models in Remote Sensing
- Image Classification Using CNN-QNN Hybrid Model with Optimized Correlated Features
- NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning
- SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors
- Representation Learning via Non-Contrastive Mutual Information
- Simple Graph Contrastive Learning via Fractional-order Neural Diffusion Networks
- SnapPix: Efficient-Coding--Inspired In-Sensor Compression for Edge Vision
- Variational Self-Supervised Learning
- PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models
- Curia-MAE: Multi-Modal Multi-Anatomy MAE Pre-Training for 3D Medical Image Segmentation
- TESSERA v2: Scaling Pixel-wise Earth Foundation Models
- MoDAl: Self-Supervised Neural Modality Discovery via Decorrelation for Speech Neuroprosthesis
- Masked Autoencoder Self Pre-Training for Defect Detection in Microelectronics
- HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentation
- FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding
- The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasound
- Measuring Déjà vu Memorization Efficiently
Related