Self-Supervised Learning with Data Augmentations Provably Isolates\n Content from Style

2021/06/08 by Julius von Kügelgen, von Kügelgen, Julius, Yash Sharma +11 · 28 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Image Processing Techniques and Applications #Image Processing and 3D Reconstruction #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2106.04619

openalex publication_date 2021/06/08 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

Self-supervised representation learning has shown remarkable success in a\nnumber of domains. A common practice is to perform data augmentation via\nhand-crafted transformations intended to leave the semantics of the data\ninvariant. We seek to understand the empirical success of this approach from a\ntheoretical perspective. We formulate the augmentation process as a latent\nvariable model by postulating a partition of the latent representation into a\ncontent component, which is assumed invariant to augmentation, and a style\ncomponent, which is allowed to change. Unlike prior work on disentanglement and\nindependent component analysis, we allow for both nontrivial statistical and\ncausal dependencies in the latent space. We study the identifiability of the\nlatent representation based on pairs of views of the observations and prove\nsufficient conditions that allow us to identify the invariant content partition\nup to an invertible mapping in both generative and discriminative settings. We\nfind numerical simulations with dependent latent variables are consistent with\nour theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional,\nvisually complex images with rich causal dependencies, which we use to study\nthe effect of data augmentations performed in practice.\n

Citations

• Auto-Encoding Variational Bayes
• Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources
• An Information-Maximization Approach to Blind Separation and Blind Deconvolution
• Independent component analysis: algorithms and applications
• Representation Learning: A Review and New Perspectives
• Learning deep representations by mutual information estimation and maximization
• Causality for Machine Learning
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder
• Shortcut learning in deep neural networks
• Towards causal generative scene models via competition of experts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Predicting What You Already Know Helps: Provable Self-Supervised\n Learning
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Contrastive Multiview Coding
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Toward Causal Representation Learning
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Exploring Simple Siamese Representation Learning

Cited by

• From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
• Learning by Analogy: A Causal Framework for Composition Generalization
• Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality
• Towards Reliable Test-Time Adaptation: Style Invariance as a Correctness Likelihood
• Self-Supervised Dynamical System Representations for Physiological Time-Series
• PISA: Prioritized Invariant Subgraph Aggregation
• Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens
• Debiasing Reward Models by Representation Learning with Guarantees
• Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions
• Thought Communication in Multiagent Collaboration
• CytoNet: A Foundation Model for the Human Cerebral Cortex at Cellular Resolution
• Towards Identifiability of Hierarchical Temporal Causal Representation Learning
• Online Time Series Forecasting with Theoretical Guarantees
• Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability
• CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations
• Towards Self-Refinement of Vision-Language Models with Triangular Consistency
• Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
• On the Identifiability of Latent Action Policies
• Nonparametric Identification of Latent Concepts
• Disentanglement of Variations with Multimodal Generative Modeling
• Mechanistic Independence: A Principle for Identifiable Disentangled Representations
• ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow
• Towards Causal Representation Learning with Observable Sources as Auxiliaries
• PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits
• Investigating Location-Regularised Self-Supervised Feature Learning for Seafloor Visual Imagery
• Contrastive Representation Learning with Trainable Augmentation Channel
• Learning Robust Intervention Representations with Delta Embeddings
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees

Related

• Self-Supervised Disentanglement by Leveraging Structure in Data Augmentations
• Stable and Causal Inference for Discriminative Self-supervised Deep Visual Representations
• Representation Learning via Invariant Causal Mechanisms
• Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation
• A Probabilistic Model Behind Self-Supervised Learning