HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations

2025/06/11 by Federici, Marco, Del Chiaro, Riccardo, van Breugel, Boris +2 · 3 citations
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.09932

Abstract

Diffusion models represent the cutting edge in image generation, but their high memory and computational demands hinder deployment on resource-constrained devices. Post-Training Quantization (PTQ) offers a promising solution by reducing the bitwidth of matrix operations. However, standard PTQ methods struggle with outliers, and achieving higher compression often requires transforming model weights and activations before quantization. In this work, we propose HadaNorm, a novel linear transformation that extends existing approaches by both normalizing channels activations and applying Hadamard transforms to effectively mitigate outliers and enable aggressive activation quantization. We demonstrate that HadaNorm consistently reduces quantization error across the various components of transformer blocks, outperforming state-of-the-art methods.

Cited by

• TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search
• ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
• STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization

Related

• HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
• CLQ: Cross-Layer Guided Orthogonal-based Quantization for Diffusion Transformers
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
• KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
• An Analysis on Quantizing Diffusion Transformers