Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

2021/06/08 by Julius von Kügelgen, von Kügelgen, Julius, Yash Sharma +11 · 41 citations
Computer Science · Engineering · Mathematics · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Processing Techniques and Applications #Image Processing and 3D Reconstruction #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.AI #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2106.04619

NeurIPS 2021 final camera-ready revision (with minor corrections)

openalex publication_date 2021/06/08 · arxiv created 2022/01/14 · arxiv updated 2022/01/17 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

Citations

• Auto-Encoding Variational Bayes
• Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources
• An Information-Maximization Approach to Blind Separation and Blind Deconvolution
• Independent component analysis: algorithms and applications
• Representation Learning: A Review and New Perspectives
• Learning deep representations by mutual information estimation and maximization
• Causality for Machine Learning
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• Shortcut learning in deep neural networks
• Towards causal generative scene models via competition of experts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Contrastive Multiview Coding
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Toward Causal Representation Learning
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Exploring Simple Siamese Representation Learning

Cited by

• From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
• Learning by Analogy: A Causal Framework for Composition Generalization
• Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality
• Towards Reliable Test-Time Adaptation: Style Invariance as a Correctness Likelihood
• Self-Supervised Dynamical System Representations for Physiological Time-Series
• PISA: Prioritized Invariant Subgraph Aggregation
• Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens
• Debiasing Reward Models by Representation Learning with Guarantees
• Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions
• Thought Communication in Multiagent Collaboration
• CytoNet: A Foundation Model for the Human Cerebral Cortex at Cellular Resolution
• Towards Identifiability of Hierarchical Temporal Causal Representation Learning
• Online Time Series Forecasting with Theoretical Guarantees
• Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability
• CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency
• Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
• On the Identifiability of Latent Action Policies
• Nonparametric Identification of Latent Concepts
• Disentanglement of Variations with Multimodal Generative Modeling
• Mechanistic Independence: A Principle for Identifiable Disentangled Representations
• ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow
• Towards Causal Representation Learning with Observable Sources as Auxiliaries
• PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits
• Investigating Location-Regularised Self-Supervised Feature Learning for Seafloor Visual Imagery
• Contrastive Representation Learning with Trainable Augmentation Channel
• Learning Robust Intervention Representations with Delta Embeddings
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees
• Incorporating Interventional Independence Improves Robustness against Interventional Distribution Shift
• When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
• Diffusion Counterfactual Generation with Semantic Abduction
• Identifiable Object Representations under Spatial Ambiguities
• Causal Representation Learning with Observational Grouping for CXR Classification
• The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations
• PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization
• Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning
• When Does LeJEPA Learn a World Model?
• You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences
• Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions
• Towards Interpretable Deep Generative Models via Causal Representation Learning
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning

Related

• Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations
• Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representations
• Representation Learning via Invariant Causal Mechanisms
• Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation
• A Probabilistic Model Behind Self-Supervised Learning