D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

2021/06/12 by Abhishek Sinha, Sinha, Abhishek, Jiaming Song +5 · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Cancer-related molecular mechanisms research #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #cs.AI #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2106.06819

arxiv created 2021/06/12 · openalex publication_date 2021/06/12 · arxiv updated 2021/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.

Citations

• NICE: Non-linear Independent Components Estimation
• Training Products of Experts by Minimizing Contrastive Divergence
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• ImageNet classification with deep convolutional neural networks
• Stable signal recovery from incomplete and inaccurate measurements
• Generative Adversarial Text to Image Synthesis
• Density estimation using Real NVP
• Adversarial Feature Learning
• Learning in Implicit Generative Models
• Towards Deeper Understanding of Variational Autoencoding Models
• VAE with a VampPrior
• InfoVAE: Information Maximizing Variational Autoencoders
• Wasserstein Auto-Encoders
• Spectral Normalization for Generative Adversarial Networks
• Distribution Matching in Variational Inference
• Neural Ordinary Differential Equations
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Learning Controllable Fair Representations
• Diagnosing and Enhancing VAE Models
• Implicit Generation and Generalization in Energy-Based Models
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Deep Residual Learning for Image Recognition
• Generative Modeling by Estimating Gradients of the Data Distribution
• Neural Discrete Representation Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Jukebox: A Generative Model for Music
• Language Models are Few-Shot Learners
• Bootstrap your own latent: A new approach to self-supervised Learning
• Denoising Diffusion Probabilistic Models
• Permutation Invariant Graph Generation via Score-Based Generative Modeling
• Denoising Diffusion Implicit Models
• Exploring Simple Siamese Representation Learning
• Score-Based Generative Modeling through Stochastic Differential Equations
• Zero-Shot Text-to-Image Generation
• Learning Transferable Visual Models From Natural Language Supervision
• Generative Adversarial Transformers
• Symbolic Music Generation with Diffusion Models
• Diffusion Models Beat GANs on Image Synthesis

Cited by

• Simulating the Visual World with Artificial Intelligence: A Roadmap
• Universal Few-Shot Spatial Control for Diffusion Models
• CoVAE: Consistency Training of Variational Autoencoders
• Text to Image for Multi-Label Image Recognition with Joint Prompt-Adapter Learning
• DiffuMural: Restoring Dunhuang Murals with Multi-scale Diffusion

Related

• Conditional Image Synthesis with Diffusion Models: A Survey
• Conditional Generation from Unconditional Diffusion Models using Denoiser Representations
• Denoising Diffusion Autoencoders are Unified Self-supervised Learners
• CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation
• Diffusion Models as Masked Autoencoders