SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenization

2025/10/06 by Théophane Vallaeys, Jakob Verbeek, Vallaeys, Théophane +3 · 3 citations
Computer Science · #Advanced Data Compression Techniques #Advanced Steganography and Watermarking Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image and Video Stabilization

paper · pdf · doi:10.48550/arxiv.2510.04961

openalex publication_date 2025/10/06 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Tokenizers are a key component of state-of-the-art generative image models, extracting the most important features from the signal while reducing data dimension and redundancy. Most current tokenizers are based on KL-regularized variational autoencoders (KL-VAE), trained with reconstruction, perceptual and adversarial losses. Diffusion decoders have been proposed as a more principled alternative to model the distribution over images conditioned on the latent. However, matching the performance of KL-VAE still requires adversarial losses, as well as a higher decoding time due to iterative sampling. To address these limitations, we introduce a new pixel diffusion decoder architecture for improved scaling and training stability, benefiting from transformer components and GAN-free training. We use distillation to replicate the performance of the diffusion decoder in an efficient single-step decoder. This makes SSDD the first diffusion decoder optimized for single-step reconstruction trained without adversarial losses, reaching higher reconstruction quality and faster sampling than KL-VAE. In particular, SSDD improves reconstruction FID from 0.87 to 0.46 with 1.4× higher throughput and preserve generation quality of DiTs with 3.8× faster sampling. As such, SSDD can be used as a drop-in replacement for KL-VAE, and for building higher-quality and faster generative models.

Citations

• DGAE: Diffusion-Guided Autoencoder for Efficient Latent Representation Learning
• Mean Flows for One-step Generative Modeling
• Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization
• FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
• Masked Autoencoders Are Effective Tokenizers for Diffusion Models
• Diffusion Autoencoders are Scalable Image Tokenizers
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
• SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer
• Randomized Autoregressive Visual Generation
• Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion
• Movie Gen: A Cast of Media Foundation Models
• Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
• HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think
• Epsilon-VAE: Denoising as Visual Decoding
• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion
• Autoregressive Image Generation without Vector Quantization
• Image and Video Tokenization with Binary Spherical Quantization
• An Image is Worth 32 Tokens for Reconstruction and Generation
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
• The Road Less Scheduled
• Improved Distribution Matching Distillation for Fast Image Synthesis
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
• Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
• Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers
• Photorealistic Video Generation with Diffusion Models
• One-step Diffusion with Distribution Matching Distillation
• SODA: Bottleneck Diffusion Models for Representation Learning
• Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
• Improved Techniques for Training Consistency Models
• Towards image compression with perfect realism at ultra-low bitrates
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
• Finite Scalar Quantization: VQ-VAE Made Simple
• InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
• SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
• DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
• High-Fidelity Image Compression with Score-based Generative Models
• Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks
• DINOv2: Learning Robust Visual Features without Supervision
• Consistency Models
• Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
• UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models
• Simple diffusion: End-to-end diffusion for high resolution images
• On the Importance of Noise Scheduling for Diffusion Models
• Scalable Diffusion Models with Transformers
• MAGVIT: Masked Generative Video Transformer
• On Distillation of Guided Diffusion Models
• Flow Matching for Generative Modeling
• Building Normalizing Flows with Stochastic Interpolants
• Lossy Image Compression with Conditional Diffusion Models
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Classifier-Free Diffusion Guidance
• DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder
• Elucidating the Design Space of Diffusion-Based Generative Models
• Autoregressive Image Generation using Residual Quantization
• MaskGIT: Masked Generative Image Transformer
• DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
• Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
• Palette: Image-to-Image Diffusion Models
• Vector-quantized Image Modeling with Improved VQGAN
• Diffusion Models Beat GANs on Image Synthesis
• Improved Denoising Diffusion Probabilistic Models
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
• Taming Transformers for High-Resolution Image Synthesis
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Probabilistic Models
• Reliable Fidelity and Diversity Metrics for Generative Models
• GLU Variants Improve Transformer
• Generative Modeling by Estimating Gradients of the Data Distribution
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention with Relative Position Representations
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• The Perception-Distortion Tradeoff
• Neural Discrete Representation Learning
• Attention Is All You Need
• U-Net: Convolutional Networks for Biomedical Image Segmentation
• Microsoft COCO: Common Objects in Context
• Image quality assessment: from error visibility to structural similarity
• Logistic-normal distributions:Some properties and uses

Cited by

• Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations
• Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective
• VUGEN: Visual Understanding priors for GENeration

Related

• Diffusion Autoencoders are Scalable Image Tokenizers
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
• Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
• Sample what you cant compress
• Diffusion Models Beat GANs on Image Synthesis