One-step Diffusion with Distribution Matching Distillation
2023/11/30 by Tianwei Yin, Michaël Gharbi, Richard Zhang +4 · 2 voices · 199 citations
#cs.CV
paper · pdf
Abstract
Diffusion models generate high-quality images but require dozens of forward passes. We introduce Distribution Matching Distillation (DMD), a procedure to transform a diffusion model into a one-step image generator with minimal impact on image quality. We enforce the one-step image generator match the diffusion model at distribution level, by minimizing an approximate KL divergence whose gradient can be expressed as the difference between 2 score functions, one of the target distribution and the other of the synthetic distribution being produced by our one-step generator. The score functions are parameterized as two diffusion models trained separately on each distribution. Combined with a simple regression loss matching the large-scale structure of the multi-step diffusion outputs, our method outperforms all published few-step diffusion approaches, reaching 2.62 FID on ImageNet 64x64 and 11.49 FID on zero-shot COCO-30k, comparable to Stable Diffusion but orders of magnitude faster. Utilizing FP16 inference, our model generates images at 20 FPS on modern hardware.
Cited by
- EgoPlay: Event-Triggered Video Editing for Egocentric Streams
- IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation
- MobileWan: Closing the Quality Gap for Mobile Video Diffusion
- Normalizing Trajectory Models
- AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars
- TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation
- FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models
- OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
- ScaleResfusion: Residual Rectified Flow based on Residual Vector Field
- Generative Video Compression with Adaptive Score Distillation
- Text-based Tactile Graphics Generation for the Visually Impaired
- Generalized Fine-Tuning of Diffusion Models via Stochastic Control and FBSDEs
- FiDeSR: High-Fidelity and Detail-Preserving One-Step Diffusion Super-Resolution
- StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
- Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation
- HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
- Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation
- SLIM: Semantic-based Low-bitrate Image compression for Machines by leveraging diffusion
- FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction
- End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
- Large Video Planner Enables Generalizable Robot Control
- SoFlow: Solution Flow Models for One-Step Generative Modeling
- WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
- Random-Bridges as Stochastic Transports for Generative Models
- MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives
- LongVie 2: Multimodal Controllable Ultra-Long Video World Model
- KlingAvatar 2.0 Technical Report
- Endless World: Real-Time 3D-Aware Long Video Generation
- TS-DP: Reinforcement Speculative Decoding For Temporal Adaptive Diffusion Policy Acceleration
- BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models
- JoyAvatar: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
- PersonaLive! Expressive Portrait Image Animation for Live Streaming
- AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Path
- Sharp Monocular View Synthesis in Less Than a Second
- Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
- Delay-Aware Diffusion Policy: Bridging the Observation-Execution Gap in Dynamic Tasks
- Unified Video Editing with Temporal Reasoner
- SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
- VDOT: Efficient Unified Video Creation via Optimal Transport Distillation
- USV: Unified Sparsification for Accelerating Video Diffusion Models
- HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
- ICM-SR: Image-Conditioned Manifold Regularization for Image Super-Resolution
- Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation
- Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
- VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory
- UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformers
- ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
- On the Design of One-step Diffusion via Shortcutting Flow Paths
- TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows
- RELIC: Interactive Video World Model with Long-Horizon Memory
- SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control
- Glance: Accelerating Diffusion Models with 1 Sample
- Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield
- SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation
- Cosine-Similarity Methods for Efficient Training and Sampling in High-Dimensional Latent Spaces
- Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model
- BIFE: Better Interaction, Fewer Errors for Minute-Long Video Generation
- Adversarial Flow Models
- Designing Instance-Level Sampling Schedules via REINFORCE with James-Stein Shrinkage
- MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Training
- MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devices
- CDLM: Consistency Diffusion Language Models For Faster Sampling
- Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learning
- FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers
- BRIC: Bridging Kinematic Plans and Physical Control at Test Time
- Block Cascading: Training Free Acceleration of Block-Causal Video Models
- CREward: A Type-Specific Creativity Reward Model
- Flow Map Distillation Without Data
- Understanding, Accelerating, and Improving MeanFlow Training
- FlowSteer: Guiding Few-Step Image Synthesis with Authentic Trajectories
- Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer
- CoD: A Diffusion Foundation Model for Image Compression
- Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation
- EvDiff: High Quality Video with an Event Camera
- One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution
- Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation
- Generative Augmented Reality: Paradigms, Technologies, and Future Applications
- AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
- Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning
- InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior
- Distribution Matching Distillation Meets Reinforcement Learning
- Functional Mean Flow in Hilbert Space
- Learning Straight Flows: Variational Flow Matching for Efficient Generation
- Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
- Null-Space Diffusion Distillation Unlocks Speed, Fidelity and Realism in Lensless Imaging
- Fast Data Attribution for Text-to-Image Models
- Hierarchical Schedule Optimization for Fast and Robust Diffusion Model Sampling
- Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
- SASG-DA: Sparse-Aware Semantic-Guided Diffusion Augmentation For Myoelectric Gesture Recognition
- MotionStream: Real-Time Video Generation with Interactive Motion Controls
- StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation
- Test-Time Iterative Error Correction for Efficient Diffusion Models
- Neodragon: Mobile Video Generation using Diffusion Transformer
- KLASS: KL-Guided Fast Inference in Masked Diffusion Models
- Towards One-step Causal Video Generation via Adversarial Self-Distillation
- Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals
- E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources
- Hybrid Consistency Policy: Decoupling Multi-Modal Diversity and Real-Time Efficiency in Robotic Manipulation
- LeMiCa: Lexicographic Minimax Path Caching for Efficient Diffusion-Based Video Generation
- Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
- Input-Aware Sparse Attention for Real-Time Co-Speech Video Generation
- DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
- Amortized Moment Matching for Visual Generation
- TridentServe: A Stage-level Serving System for Diffusion Pipelines
- High-Fidelity Two-Step Image Generation via Teacher-Aligned End-to-End Distillation
- Generative Modeling via Drifting
- Balanced conic rectified flow
- Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Sampling
- FARMER: Flow AutoRegressive Transformer over Pixels
- Improved Training Technique for Shortcut Models
- Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation
- Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
- AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
- AlphaFlow: Understanding and Improving MeanFlow Models
- SCEESR: Semantic-Control Edge Enhancement for Diffusion-Based Super-Resolution
- Imitation Learning Policy based on Multi-Step Consistent Integration Shortcut Model
- GAS: Improving Discretization of Diffusion ODEs via Generalized Adversarial Solver
- Adaptive Discretization for Consistency Models
- One-step Diffusion Models with Bregman Density Ratio Matching
- Learning an Image Editing Model without Image Editing Pairs
- pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation
- Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch
- FlashWorld: High-quality 3D Scene Generation within Seconds
- FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution
- Hierarchical Koopman Diffusion: Fast Generation with Interpretable Diffusion Trajectory
- Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
- Fine-Tuning Flow Matching via Maximum Likelihood Estimation of Reconstructions
- DreamX-World 1.0: A General-Purpose Interactive World Model
- One Pass Is Not Enough: Recursive Latent Refinement for Generative Models
- Real-Time Motion-Controllable Autoregressive Video Diffusion
- Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
- LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
- Fine-grained Defocus Blur Control for Generative Image Models
- Leveraging Generative AI for large-scale prediction-based networking
- SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization
- Knowledge Distillation Detection for Open-weights Models
- Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation
- LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation
- LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration
- Audio Driven Real-Time Facial Animation for Social Telepresence
- Riemannian Consistency Model
- Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangents
- UCD: Unconditional Discriminator Promotes Nash Equilibrium in GANs
- InfVSR: Toward Consistency-Driven Streaming Generative Video Super-Resolution
- Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
- F-scheduler: illuminating the free-lunch design space for fast sampling of diffusion models
- Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
- Score Distillation of Flow Matching Models
- CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Models
- Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
- DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space
- Autoregressive Video Generation beyond Next Frames Prediction
- Towards Redundancy Reduction in Diffusion Models for Efficient Video Super-Resolution
- Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
- MeanFlowSE: One-Step Generative Speech Enhancement via MeanFlow
- Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
- LongLive: Real-time Interactive Long Video Generation
- Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
- FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing
- Consistency Models as Plug-and-Play Priors for Inverse Problems
- Score-based Idempotent Distillation of Diffusion Models
- SD3.5-Flash: Distribution-Guided Distillation of Generative Flows
- ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
- Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation
- A Gradient Flow Approach to Solving Inverse Problems with Latent Diffusion Models
- OS-DiffVSR: Towards One-step Latent Diffusion Model for High-detailed Real-world Video Super-Resolution
- HERO: Hierarchical Extrapolation and Refresh for Efficient World Models
- DiffusionNFT: Online Diffusion Reinforcement with Forward Process
- LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition
- Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling
- SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation
- Enhancing Physical Consistency in Lightweight World Models
- Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
- Transition Models: Rethinking the Generative Learning Objective
- Q-Sched: Pushing the Boundaries of Few-Step Diffusion Models with Quantization-Aware Scheduling
- Acoustic Interference Suppression in Ultrasound images for Real-Time HIFU Monitoring Using an Image-Based Latent Diffusion Model
- Phased One-Step Adversarial Equilibrium for Video Diffusion Models
- StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimation
- Scaling Group Inference for Diverse and High-Quality Generation
- CurveFlow: Curvature-Guided Flow Matching for Image Generation
- CTA-Flux: Integrating Chinese Cultural Semantics into High-Quality English Text-to-Image Communities
- DiffIER: Optimizing Diffusion Models with Iterative Error Reduction
- One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
- Cognitive Structure Generation: From Educational Priors to Policy Optimization
- Matrix-game 2.0: An open-source real-time and streaming interactive world model
- Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
- Yan: Foundational Interactive Video Generation
- Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
- FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation
- Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
- SwiftVideo: A Unified Framework for Few-Step Video Generation through Trajectory-Distribution Alignment
- InstantEdit: Text-Guided Few-Step Image Editing with Piecewise Rectified Flow
- DogFit: Domain-guided Fine-tuning for Efficient Transfer Learning of Diffusion Models
- Diffusion Once and Done: Degradation-Aware LoRA for Efficient All-in-One Image Restoration
- READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation
- DisCo3D: Distilling Multi-View Consistency for 3D Scene Editing
- Sortblock: Similarity-Aware Feature Reuse for Diffusion Model
- ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning
- MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
Discussions
Related