TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillation

2023/03/07 by David Berthelot, Berthelot, David, Arnaud Autef +14 · 13 citations
Computer Science · Medicine · #Advanced Mathematical Modeling in Engineering #Advanced Neuroimaging Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #MRI in cancer diagnosis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2303.04248

openalex publication_date 2023/03/07 · openalex created_date 2023/03/10 · openalex updated_date 2026/07/28

Abstract

Denoising Diffusion models have demonstrated their proficiency for generative sampling. However, generating good samples often requires many iterations. Consequently, techniques such as binary time-distillation (BTD) have been proposed to reduce the number of network calls for a fixed architecture. In this paper, we introduce TRAnsitive Closure Time-distillation (TRACT), a new method that extends BTD. For single step diffusion,TRACT improves FID by up to 2.4x on the same architecture, and achieves new single-step Denoising Diffusion Implicit Models (DDIM) state-of-the-art FID (7.4 for ImageNet64, 3.8 for CIFAR10). Finally we tease apart the method through extended ablations. The PyTorch implementation will be released soon.

Cited by

• LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization
• BézierFlow: Learning Bézier Stochastic Interpolant Schedulers for Few-Step Generation
• Flow Map Distillation Without Data
• Understanding, Accelerating, and Improving MeanFlow Training
• Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning
• Towards One-step Causal Video Generation via Adversarial Self-Distillation
• Adaptive Stochastic Coefficients for Accelerating Diffusion Sampling
• AlphaFlow: Understanding and Improving MeanFlow Models
• Adaptive Discretization for Consistency Models
• Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangents
• CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Models
• VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation

Related

• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping
• Directly Denoising Diffusion Models
• Scale-wise Distillation of Diffusion Models
• Think While You Generate: Discrete Diffusion with Planned Denoising
• Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment