Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation

2023/07/05 by Sébastien Lachapelle, Lachapelle, Sébastien, Divyat Mahajan +5 · 16 citations
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Cartesian product #Computer science #Discrete mathematics #Domain Adaptation and Few-Shot Learning #Extrapolation #FOS: Computer and information sciences #Feature learning #Generative Adversarial Networks and Image Synthesis #I.2.6 #I.5.1 #Identifiability #Identification (biology) #Invertible matrix #Latent class model #Latent variable #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine learning #Mathematics #Object (grammar) #Pure mathematics #Representation (politics) #Statistics #Theoretical computer science

paper · pdf · doi:10.48550/arxiv.2307.02598

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/07/05 · openalex created_date 2023/07/08 · openalex updated_date 2026/07/28

Abstract

We tackle the problems of latent variables identification and ``out-of-support'' image generation in representation learning. We show that both are possible for a class of decoders that we call additive, which are reminiscent of decoders used for object-centric representation learning (OCRL) and well suited for images that can be decomposed as a sum of object-specific images. We provide conditions under which exactly solving the reconstruction problem using an additive decoder is guaranteed to identify the blocks of latent variables up to permutation and block-wise invertible transformations. This guarantee relies only on very weak assumptions about the distribution of the latent factors, which might present statistical dependencies and have an almost arbitrarily shaped support. Our result provides a new setting where nonlinear independent component analysis (ICA) is possible and adds to our theoretical understanding of OCRL methods. We also show theoretically that additive decoders can generate novel images by recombining observed factors of variations in novel ways, an ability we refer to as Cartesian-product extrapolation. We show empirically that additivity is crucial for both identifiability and extrapolation on simulated data.

Cited by

• Beyond ICA: Identifiability by Symmetry Breaking
• From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
• Learning by Analogy: A Causal Framework for Composition Generalization
• Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality
• Is Generation Required for Data-Efficient Perception?
• Operationalizing Quantized Disentanglement
• Disentangled Representation Learning via Modular Compositional Bias
• Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability
• Provable Affine Identifiability of Nonlinear CCA under Latent Distributional Priors
• Nonparametric Identification of Latent Concepts
• Mechanistic Independence: A Principle for Identifiable Disentangled Representations
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees
• Identifiable Object Representations under Spatial Ambiguities
• Learning Joint Interventional Effects from Single-Variable Interventions in Additive Models
• unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning
• Extrapolation Guarantees for Perturbation Modeling Under the Additive Latent Shift Assumption

Related

• Towards efficient representation identification in supervised learning
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Representation Learning Through Latent Canonicalizations
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Towards Causal Representation Learning with Observable Sources as Auxiliaries