Bilateral Denoising Diffusion Models

2021/08/26 by Max W. Y. Lam, Lam, Max W. Y., Jun Wang +7 · 2 citations
Computer Science · Engineering · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Audio and Speech Processing (eess.AS) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Music and Audio Processing #Signal Processing (eess.SP) #Sound (cs.SD) #cs.AI #cs.LG #cs.SD #eess.AS #eess.SP #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2108.11514

openalex publication_date 2021/08/26 · arxiv created 2021/09/14 · arxiv updated 2021/09/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Denoising diffusion probabilistic models (DDPMs) have emerged as competitive generative models yet brought challenges to efficient sampling. In this paper, we propose novel bilateral denoising diffusion models (BDDMs), which take significantly fewer steps to generate high-quality samples. From a bilateral modeling objective, BDDMs parameterize the forward and reverse processes with a score network and a scheduling network, respectively. We show that a new lower bound tighter than the standard evidence lower bound can be derived as a surrogate objective for training the two networks. In particular, BDDMs are efficient, simple-to-train, and capable of further improving any pre-trained DDPM by optimizing the inference noise schedules. Our experiments demonstrated that BDDMs can generate high-fidelity samples with as few as 3 sampling steps and produce comparable or even higher quality samples than DDPMs using 1000 steps with only 16 sampling steps (a 62x speedup).

Citations

• Generative Adversarial Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• A Connection Between Score Matching and Denoising Autoencoders
• Training Products of Experts by Minimizing Contrastive Divergence
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Density estimation using Real NVP
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Language Models are Few-Shot Learners
• Improved Techniques for Training Score-Based Generative Models
• Denoising Diffusion Probabilistic Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Denoising Diffusion Implicit Models
• Learning Multiple Layers of Features from Tiny Images
• Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
• Improved Denoising Diffusion Probabilistic Models
• Noise Estimation for Generative Diffusion Models
• Diffusion Models Beat GANs on Image Synthesis
• On Fast Sampling of Diffusion Probabilistic Models

Cited by

• DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
• BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Related

• Improved Denoising Diffusion Probabilistic Models
• Denoising Diffusion Implicit Models
• Semi-Implicit Denoising Diffusion Models (SIDDMs)
• Denoising Diffusion Probabilistic Models in Six Simple Steps
• The Missing U for Efficient Diffusion Models