Conditional Image Generation by Conditioning Variational Auto-Encoders

2021/02/24 by William Harvey, William R. Harvey, Saeid Naderiparizi +4 · 6 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Cell Image Analysis Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Model Reduction and Neural Networks #cs.AI #cs.CV

paper · pdf · doi:10.48550/arxiv.2102.12037

37 pages, 20 figures

openalex publication_date 2021/02/24 · arxiv created 2022/05/28 · arxiv updated 2022/05/31 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28

Abstract

We present a conditional variational auto-encoder (VAE) which, to avoid the substantial cost of training from scratch, uses an architecture and training objective capable of leveraging a foundation model in the form of a pretrained unconditional VAE. To train the conditional VAE, we only need to train an artifact to perform amortized inference over the unconditional VAE's latent variables given a conditioning input. We demonstrate our approach on tasks including image inpainting, for which it outperforms state-of-the-art GAN-based approaches at faithfully representing the inherent uncertainty. We conclude by describing a possible application of our inpainting model, in which it is used to perform Bayesian experimental design for the purpose of guiding a sensor.

Cited by

• One-shot Conditional Sampling: MMD meets Nearest Neighbors
• Humanoid World Models: Open World Foundation Models for Humanoid Robotics
• Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding
• A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications
• DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusion
• Fairness in Machine Learning-based Hand Load Estimation: A Case Study on Load Carriage Tasks

Related

• CDVAE: Co-embedding Deep Variational Auto Encoder for Conditional Variational Generation
• Improving Conditional VAE with approximation using Normalizing Flows
• Variational Autoencoder with Arbitrary Conditioning
• Variational Conditional GAN for Fine-grained Controllable Image Generation
• Generating new pictures in complex datasets with a simple neural network