ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers

2025/12/03 by Huang, Feice, Han, Zuliang, Zhou Xing +6
Computer Science · Medicine · #Advanced Neural Network Applications #Advanced Neuroimaging Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2512.03673

openalex publication_date 2025/12/03 · openalex created_date 2025/12/05 · openalex updated_date 2026/07/28

Abstract

Diffusion transformers have demonstrated strong capabilities in generating high-quality images. However, as model size increases, the growing memory footprint and inference latency pose significant challenges for practical deployment. Recent studies in large language models (LLMs) show that rotation-based techniques can smooth outliers and enable 4-bit quantization, but these approaches often incur substantial overhead and struggle with row-wise outliers in diffusion transformers. To address these challenges, we propose ConvRot, a group-wise rotation-based quantization method that leverages regular Hadamard transform (RHT) to suppress both row-wise and column-wise outliers while reducing complexity from quadratic to linear. Building on this, we design ConvLinear4bit, a plug-and-play module that integrates rotation, quantization, GEMM, and dequantization, enabling W4A4 inference without retraining and preserving visual quality. Experiments on FLUX.1-dev demonstrate a 2.26× speedup and 4.05× memory reduction while maintaining image fidelity. To our knowledge, this is the first application of rotation-based quantization for plug-and-play W4A4 inference in diffusion transformers.

Citations

• Qwen-Image Technical Report
• HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
• HadaCore: Tensor Core Accelerated Hadamard Transform Kernel
• HunyuanVideo: A Systematic Framework For Large Video Generative Models
• Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
• SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
• Dynamic Diffusion Transformer
• ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
• MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization
• SpinQuant: LLM quantization with learned rotations
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
• One-step Diffusion with Distribution Matching Distillation
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
• A Survey on Model Compression for Large Language Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• QLoRA: Efficient Finetuning of Quantized LLMs
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• Q-Diffusion: Quantizing Diffusion Models
• Scalable Diffusion Models with Transformers
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
• High-Resolution Image Synthesis with Latent Diffusion Models
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models

Related

• HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
• FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers
• KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
• CLQ: Cross-Layer Guided Orthogonal-based Quantization for Diffusion Transformers