SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher

2024/08/26 by Trung Dao, Dao, Trung, Thuan Hoang Nguyen +11 · 21 citations
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Scientific Computing and Data Management #Semantic Web and Ontologies #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2408.14176

openalex publication_date 2024/08/26 · openalex created_date 2024/09/21 · openalex updated_date 2026/07/28

Abstract

In this paper, we aim to enhance the performance of SwiftBrush, a prominent one-step text-to-image diffusion model, to be competitive with its multi-step Stable Diffusion counterpart. Initially, we explore the quality-diversity trade-off between SwiftBrush and SD Turbo: the former excels in image diversity, while the latter excels in image quality. This observation motivates our proposed modifications in the training methodology, including better weight initialization and efficient LoRA training. Moreover, our introduction of a novel clamped CLIP loss enhances image-text alignment and results in improved image quality. Remarkably, by combining the weights of models trained with efficient LoRA and full training, we achieve a new state-of-the-art one-step diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and multi-step Stable Diffusion models. The project page is available at https://swiftbrushv2.github.io.

Cited by

• Diversity Has Always Been There in Your Visual Autoregressive Models
• Improved Training Technique for Shortcut Models
• Adversarial Concept Distillation for One-Step Diffusion Personalization
• One-step Diffusion Models with Bregman Density Ratio Matching
• Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings
• FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing
• SD3.5-Flash: Distribution-Guided Distillation of Generative Flows
• Taming Diffusion Transformer for Efficient Mobile Video Generation in Seconds
• CSD-VAR: Content-Style Decomposition in Visual Autoregressive Models
• FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency
• FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation
• From Cradle to Cane: A Two-Pass Framework for High-Fidelity Lifespan Face Aging
• Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability
• One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models
• MIND-Edit: MLLM Insight-Driven Editing via Language-Vision Projection
• Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse
• One-Step Diffusion-Based Image Compression with Semantic Distillation
• Few-Step Diffusion via Score identity Distillation
• Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform
• ChordVideo: One-Step, Training-Free, Temporally Consistent Video Editing via Low-Energy Transport

Related

• SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation
• InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
• HiPA: Enabling One-Step Text-to-Image Diffusion Models via High-Frequency-Promoting Adaptation
• Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation
• SpeedUpNet: A Plug-and-Play Adapter Network for Accelerating Text-to-Image Diffusion Models