Turning LLM Activations Quantization-Friendly

2025/05/11 by Patrik Czakó, Czakó, Patrik, Gábor Kertész +3 · 1 citation
Computer Science · Engineering · Physics and Astronomy · #Advancements in Photolithography Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Magnetic confinement fusion research #Mathematics, Computing, and Information Processing #Metric (unit) #Outlier #Quantization (signal processing) #Scaling #Smoothing #Spurious relationship

paper · pdf · doi:10.48550/arxiv.2506.01967

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/11 · openalex created_date 2025/10/14 · openalex updated_date 2026/08/05

Abstract

Quantization effectively reduces the serving costs of Large Language Models (LLMs) by speeding up data movement through compressed parameters and enabling faster operations via integer arithmetic. However, activating integer arithmetic requires quantizing both weights and activations, which poses challenges due to the significant outliers in LLMs that increase quantization error. In this work, we investigate these outliers with an emphasis on their effect on layer-wise quantization error, then examine how smoothing and rotation transform the observed values. Our primary contributions include introducing a new metric to measure and visualize quantization difficulty based on channel magnitudes, as well as proposing a hybrid approach that applies channel-wise scaling before rotation, supported by a mathematical formulation of its benefits.

Citations

• FlatQuant: Flatness Matters for LLM Quantization
• The Llama 3 Herd of Models
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
• SpinQuant: LLM quantization with learned rotations
• Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• Model Compression and Efficient Inference for Large Language Models: A Survey
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
• Mistral 7B
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• A Survey on Model Compression for Large Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• SqueezeLLM: Dense-and-Sparse Quantization
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Pointer Sentinel Mixture Models

Cited by

• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs

Related

• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
• Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
• Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models
• Why Do Some Inputs Break Low-Bit LLM Quantization?
• OutlierTune: Efficient Channel-Wise Quantization for Large Language Models