Representation Learning via Invariant Causal Mechanisms

2020/10/15 by Jovana Mitrovic, Mitrovic, Jovana, Brian McWilliams +7 · 8 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2010.07922

openalex publication_date 2020/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success of these methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on 51 out of 57 games.

Citations

• ImageNet Large Scale Visual Recognition Challenge
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Cited by

• Towards Reliable Test-Time Adaptation: Style Invariance as a Correctness Likelihood
• Structured Contrastive Learning for Interpretable Latent Representations
• An Augmentation Overlap Theory of Contrastive Learning
• Evolving Graph Learning for Out-of-Distribution Generalization in Non-stationary Environments
• Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals
• Minimal Semantic Sufficiency Meets Unsupervised Domain Generalization
• Relating Graph Neural Networks to Structural Causal Models
• Decomposing and Revising What Language Models Generate

Related

• Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representations
• Generalizable Information Theoretic Causal Representation
• Invariant Causal Mechanisms through Distribution Matching
• Self-Supervised Learning with Data Augmentations Provably Isolates\n Content from Style
• Improving Self-Supervised Learning by Characterizing Idealized Representations