Improved Distribution Matching Distillation for Fast Image Synthesis
2024/05/23 by Tianwei Yin, Michaël Gharbi, Yin, Tianwei +11 · 217 citations
Computer Science · Engineering · #Advanced Image and Video Retrieval Techniques #Image Processing Techniques and Applications #Advanced Vision and Imaging
paper · pdf · doi:10.48550/arxiv.2405.14867
Abstract
Recent approaches have shown promises distilling diffusion models into efficient one-step generators. Among them, Distribution Matching Distillation (DMD) produces one-step generators that match their teacher in distribution, without enforcing a one-to-one correspondence with the sampling trajectories of their teachers. However, to ensure stable training, DMD requires an additional regression loss computed using a large set of noise-image pairs generated by the teacher with many steps of a deterministic sampler. This is costly for large-scale text-to-image synthesis and limits the student's quality, tying it too closely to the teacher's original sampling paths. We introduce DMD2, a set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. We show that the resulting instability is due to the fake critic not estimating the distribution of generated samples accurately and propose a two time-scale update rule as a remedy. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images. This lets us train the student model on real data, mitigating the imperfect real score estimation from the teacher model, and enhancing quality. Lastly, we modify the training procedure to enable multi-step sampling. We identify and address the training-inference input mismatch problem in this setting, by simulating inference-time generator samples during training time. Taken together, our improvements set new benchmarks in one-step image generation, with FID scores of 1.28 on ImageNet-64x64 and 8.35 on zero-shot COCO 2014, surpassing the original teacher despite a 500X reduction in inference cost. Further, we show our approach can generate megapixel images by distilling SDXL, demonstrating exceptional visual quality among few-step methods.
Cited by
- Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance
- Distribution Matching Variational AutoEncoder
- LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation
- D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
- Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion
- ReDiF: Reinforced Distillation for Few Step Diffusion
- IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation
- Self-Evaluation Unlocks Any-Step Text-to-Image Generation
- Manifold-Constrained Noise Optimization for Diverse Diffusion Sampling
- ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation
- MobileWan: Closing the Quality Gap for Mobile Video Diffusion
- Normalizing Trajectory Models
- AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars
- FlowCTS: On-policy Continuous Trajectory Supervision of Flow Models
- OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
- Generative Video Compression with Adaptive Score Distillation
- Knot Forcing: Taming Autoregressive Video Diffusion Models for Real-time Infinite Interactive Portrait Animation
- HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming
- Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation
- FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction
- Kling-Omni Technical Report
- End-to-End Training for Autoregressive Video Diffusion via Self-Resampling
- WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling
- LongVie 2: Multimodal Controllable Ultra-Long Video World Model
- Image Diffusion Preview with Consistency Solver
- Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
- Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10×
- Endless World: Real-Time 3D-Aware Long Video Generation
- Boosting Monocular Metric Depth Estimation via Bokeh Rendering
- BAgger: Backwards Aggregation for Mitigating Drift in Autoregressive Video Diffusion Models
- JoyStreamer-Flash: Real-time and Infinite Audio-Driven Avatar Generation with Autoregressive Diffusion
- PersonaLive! Expressive Portrait Image Animation for Live Streaming
- Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation
- SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
- VDOT: Efficient Unified Video Creation via Optimal Transport Distillation
- EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing
- USV: Unified Sparsification for Accelerating Video Diffusion Models
- Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillation
- Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length
- Refaçade: Editing Object with Given Reference Texture
- On the Design of One-step Diffusion via Shortcutting Flow Paths
- TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flows
- RELIC: Interactive Video World Model with Long-Horizon Memory
- Glance: Accelerating Diffusion Models with 1 Sample
- Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield
- SpriteHand: Real-Time Versatile Hand-Object Interaction with Autoregressive Video Generation
- Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model
- Adversarial Flow Models
- MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Training
- MobileI2V: Fast and High-Resolution Image-to-Video on Mobile Devices
- Flash-DMD: Towards High-Fidelity Few-Step Image Generation with Efficient Distillation and Joint Reinforcement Learning
- Block Cascading: Training Free Acceleration of Block-Causal Video Models
- CREward: A Type-Specific Creativity Reward Model
- GigaWorld-0: World Models as Data Engine to Empower Embodied AI
- Inferix: A Block-Diffusion based Next-Generation Inference Engine for World Simulation
- Flow Map Distillation Without Data
- Understanding, Accelerating, and Improving MeanFlow Training
- FlowSteer: Guiding Few-Step Image Synthesis with Authentic Trajectories
- Uni-DAD: Unified Distillation and Adaptation of Diffusion Models for Few-step Few-shot Image Generation
- EvDiff: High Quality Video with an Event Camera
- Generative Augmented Reality: Paradigms, Technologies, and Future Applications
- AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers
- SkyReels-Text: Fine-Grained Font-Controllable Text Editing for Poster Design
- Distribution Matching Distillation Meets Reinforcement Learning
- Learning Straight Flows: Variational Flow Matching for Efficient Generation
- Fast Data Attribution for Text-to-Image Models
- Noise Conditional Variational Score Distillation
- Generative AI Meets 6G and Beyond: Diffusion Models for Semantic Communications
- MotionStream: Real-Time Video Generation with Interactive Motion Controls
- StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation
- Test-Time Iterative Error Correction for Efficient Diffusion Models
- Towards One-step Causal Video Generation via Adversarial Self-Distillation
- Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals
- LeMiCa: Lexicographic Minimax Path Caching for Efficient Diffusion-Based Video Generation
- Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation
- SeFi-Image: A Text-to-Image Foundation Model with Semantic-First Diffusion
- DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation
- Amortized Moment Matching for Visual Generation
- Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
- High-Fidelity Two-Step Image Generation via Teacher-Aligned End-to-End Distillation
- Alice v1: Distillation-Enhanced Video Generation Surpassing Closed-Source Models
- Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Sampling
- Improved Training Technique for Shortcut Models
- Blockwise Flow Matching: Improving Flow Matching Models For Efficient High-Quality Generation
- Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation
- Adversarial Concept Distillation for One-Step Diffusion Personalization
- AlphaFlow: Understanding and Improving MeanFlow Models
- GigaBrain-0: A World Model-Powered Vision-Language-Action Model
- GAS: Improving Discretization of Diffusion ODEs via Generalized Adversarial Solver
- Adaptive Discretization for Consistency Models
- One-step Diffusion Models with Bregman Density Ratio Matching
- Learning an Image Editing Model without Image Editing Pairs
- pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation
- FlashWorld: High-quality 3D Scene Generation within Seconds
- Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
- DreamX-World 1.0: A General-Purpose Interactive World Model
- Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation
- Real-Time Motion-Controllable Autoregressive Video Diffusion
- Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
- SkipSR: Faster Super Resolution with Token Skipping
- LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
- Fine-grained Defocus Blur Control for Generative Image Models
- SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization
- Knowledge Distillation Detection for Open-weights Models
- ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation
- PocketSR: The Super-Resolution Expert in Your Pocket Mobiles
- LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration
- Align Your Tangent: Training Better Consistency Models via Manifold-Aligned Tangents
- UCD: Unconditional Discriminator Promotes Nash Equilibrium in GANs
- InfVSR: Toward Consistency-Driven Streaming Generative Video Super-Resolution
- F-scheduler: illuminating the free-lunch design space for fast sampling of diffusion models
- Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
- Rolling Forcing: Autoregressive Long Video Diffusion in Real Time
- Score Distillation of Flow Matching Models
- CMT: Mid-Training for Efficient Learning of Consistency, Mean Flow, and Flow Map Models
- Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
- DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Space
- NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding
- Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
- Scale-Wise VAR is Secretly Discrete Diffusion
- LongLive: Real-time Interactive Long Video Generation
- Universal Inverse Distillation for Matching Models with Real-Data Supervision (No GANs)
- Overclocking Electrostatic Generative Models
- RAPID3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer
- Consistency Models as Plug-and-Play Priors for Inverse Problems
- Score-based Idempotent Distillation of Diffusion Models
- SD3.5-Flash: Distribution-Guided Distillation of Generative Flows
- ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
- Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation
- A Gradient Flow Approach to Solving Inverse Problems with Latent Diffusion Models
- LowDiff: Efficient Diffusion Sampling with Low-Resolution Condition
- Transition Models: Rethinking the Generative Learning Objective
- VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation
- Phased One-Step Adversarial Equilibrium for Video Diffusion Models
- Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
- BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation
- Stable Diffusion Models are Secretly Good at Visual In-Context Learning
- TaoCache: Structure-Maintained Video Generation Acceleration
- Yan: Foundational Interactive Video Generation
- Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
- FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation
- Fewer Denoising Steps or Cheaper Per-Step Inference: Towards Compute-Optimal Diffusion Model Deployment
- SwiftVideo: A Unified Framework for Few-Step Video Generation through Trajectory-Distribution Alignment
- DisCo3D: Distilling Multi-View Consistency for 3D Scene Editing
- Imbalance-Robust and Sampling-Efficient Continuous Conditional GANs via Adaptive Vicinal Learning and Auxiliary Regularization
- Sortblock: Similarity-Aware Feature Reuse for Diffusion Model
- Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds
- CHORDS: Diffusion Sampling Accelerator with Multi-core Hierarchical ODE Solvers
- Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis
- DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
- Qwen-Image-Flash: Beyond Objective Design
- Qwen-Image-2.0 Technical Report
- CSD-VAR: Content-Style Decomposition in Visual Autoregressive Models
- The Serial Scaling Hypothesis
- Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation
- FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency
- Learning few-step posterior samplers by unfolding and distillation of diffusion models
- ActionParty: Multi-Subject Action Binding in Generative Video Games
- JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
- Pyramidal Patchification Flow for Visual Generation
- VMoBA: Mixture-of-Block Attention for Video Diffusion Models
- StableCodec: Taming One-Step Diffusion for Extreme Image Compression
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion
- Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controls
- Align Your Flow: Scaling Continuous-Time Flow Map Distillation
- Contrastive Flow Matching
- SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Training
- Dual-Expert Consistency Model for Efficient and High-Quality Video Generation
- TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via Autoregressive Diffusion Models
- LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model
- SenseFlow: Scaling Distribution Matching for Flow-based Text-to-Image Distillation
- DLM-One: Diffusion Language Models for One-Step Sequence Generation
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- Uni-Instruct: One-step Diffusion Model through Unified Diffusion Divergence Instruction
- MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-on
- Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models
- Understanding Generalization in Diffusion Distillation via Probability Flow Distance
- Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling
- Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
- DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution
- One-Step Diffusion-Based Image Compression with Semantic Distillation
- Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging
- Learning to Integrate Diffusion ODEs by Averaging the Derivatives
- Few-Step Diffusion via Score identity Distillation
- VSA: Faster Video Diffusion with Trainable Sparse Attention
- One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
- DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
- MoWorld: A Flash World Model
- Fast Text-to-Audio Generation with Adversarial Post-Training
- Distilling Drifting Transformers with Representation Autoencoders
- Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention
- Distilling Two-Timed Flow Models by Separately Matching Initial and Terminal Velocities
- Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
- Helios: Real Real-Time Long Video Generation Model
- Solaris: Building a Multiplayer Video World Model in Minecraft
- WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
- BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models
- SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture
- Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation
- VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
- Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
- PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion
- HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds
- Continuous Adversarial Flow Models
- SDiT: Semantic Region-Adaptive for Diffusion Transformers
- Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging
- Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models
- Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation
- Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models
- DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging
- Wan-Animate-2: Pushing the Application Boundaries of Character Animation
- Diffusion Distillation With Direct Preference Optimization For Efficient 3D LiDAR Scene Completion
- Autoregressive Distillation of Diffusion Transformers
- ADT: Tuning Diffusion Models with Adversarial Supervision
- On Equivariance and Fast Sampling in Video Diffusion Models Trained with Warped Noise
- ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration
- Gaussian Mixture Flow Matching Models
Related