VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation

2025/08/28 by Leyang Wang, Wang, Leyang, Ming‐Tian Zhang +5
Engineering · #Advanced Control Systems Optimization #FOS: Computer and information sciences #Machine Learning (cs.LG) #Process Optimization and Integration

paper · pdf · doi:10.48550/arxiv.2508.20646

openalex publication_date 2025/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently, diffusion distillation methods have compressed thousand-step teacher diffusion models into one-step student generators while preserving sample quality. Most existing approaches train the student model using a diffusive divergence whose gradient is approximated via the student's score function, learned through denoising score matching (DSM). Since DSM training is imperfect, the resulting gradient estimate is inevitably biased, leading to sub-optimal performance. In this paper, we propose VarDiU (pronounced /va:rdju:/), a Variational Diffusive Upper Bound that admits an unbiased gradient estimator and can be directly applied to diffusion distillation. Using this objective, we compare our method with Diff-Instruct and demonstrate that it achieves higher generation quality and enables a more efficient and stable training procedure for one-step diffusion distillation.

Citations

• Inductive Moment Matching
• One-step Diffusion Models with f-Divergence Distribution Matching
• Towards Training One-Step Diffusion Models Without Distillation
• Distributional Diffusion Models with Scoring Rules
• Normalizing Flows are Capable Generative Models
• Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step
• Training Neural Samplers with Reverse Diffusive KL Divergence
• Multistep Distillation of Diffusion Models via Moment Matching
• EM Distillation for One-step Diffusion Models
• Improved Distribution Matching Distillation for Fast Image Synthesis
• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation
• Bidirectional Consistency Models
• Multistep Consistency Models
• Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration
• Improved Techniques for Training Consistency Models
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
• TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation
• Consistency Models
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• On Distillation of Guided Diffusion Models
• Flow Matching for Generative Modeling
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Flow Annealed Importance Sampling Bootstrap
• Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
• Elucidating the Design Space of Diffusion-Based Generative Models
• Pseudo Numerical Methods for Diffusion Models on Manifolds
• Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
• Diffusion Schr "odinger Bridge with Applications to Score-Based\n Generative Modeling
• Improved Denoising Diffusion Probabilistic Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Variational f-divergence Minimization
• Generative Modeling by Estimating Gradients of the Data Distribution
• Neural Spline Flows
• Variational Inference using Implicit Distributions
• Variational Inference with Normalizing Flows
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Tweedie’s Formula and Selection Bias
• A Connection Between Score Matching and Denoising Autoencoders

Related

• SFDDM: Single-fold Distillation for Diffusion models
• Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction
• One-step Diffusion Models with f-Divergence Distribution Matching
• Multi-student Diffusion Distillation for Better One-step Generators
• Towards Training One-Step Diffusion Models Without Distillation