SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

2022/11/18 by Guangxuan Xiao, Xiao, Guangxuan, Ji Lin +10 · 1 voice · 201 citations
Computer Science · #Machine Learning in Healthcare #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2211.10438

openalex publication_date 2022/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

Cited by

• Adversarial Prompts for Acceptance Collapse in Speculative Decoding
• Quantize with Confidence? An Empirical Study of Quantization for Code Generation
• Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference
• GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
• BitNet Text Embeddings
• Contraction-Gauge Preconditioning for Quantized Matrix Multiplication
• Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models
• NIRVANA: Structured Pruning Reimagined for Large Language Model Compression
• PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
• Quantizing Recursive Reasoning Models
• Do Transformers Need Three Projections? Systematic Study of QKV Variants
• Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)
• ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
• A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
• Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2
• Argus: Token Aware Distributed LLM Inference Optimization
• OptiNIC: A Resilient and Tail-Optimal RDMA NIC for Distributed ML Workloads
• ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
• Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization
• Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling
• MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
• StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
• GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
• TOGGLE: Temporal Logic-Guided Large Language Model Compression for Edge
• LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
• PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
• Per-Axis Weight Deltas for Frequent Model Updates
• VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse
• SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models
• Arithmetic-Intensity-Aware Quantization
• HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
• SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
• CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving
• ELANA: A Simple Energy and Latency Analyzer for LLMs
• Optimizing LLMs Using Quantization for Mobile Execution
• Quantization Blindspots: How Model Compression Breaks Backdoor Defenses
• HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models
• SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
• SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
• ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
• KVNAND: Efficient On-Device Large Language Model Inference Using DRAM-Free In-Flash Computing
• UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
• Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models
• VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
• Feed-Forward 3D Gaussian Splatting Compression with Long-Context Modeling
• HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
• TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
• Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challenges
• CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
• DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection
• VLM in a flash: I/O-Efficient Sparsification of Vision-Language Model via Neuron Chunking
• QuantKAN: A Unified Quantization Framework for Kolmogorov Arnold Networks
• Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration
• Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
• Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
• R2Q: Towards Robust 2-Bit Large Language Models via Residual Refinement Quantization
• E³-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models
• Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
• Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
• Quant-Trim in Practice: Improved Cross-Platform Low-Bit Deployment on Edge NPUs
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
• Dissecting and Re-architecting 3D NAND Flash PIM Arrays for Efficient Single-Batch Token Generation in LLMs
• Efficiently Training A Flat Neural Network Before It has been Quantizated
• ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
• P3-LLM: An Integrated NPU-PIM Accelerator for LLM Inference Using Hybrid Numerical Formats
• You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
• Block Rotation is All You Need for MXFP4 Quantization
• DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control
• FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning
• STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inference
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
• RDQ: Residual Distribution Quantization for Large Language Models
• HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
• Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
• QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
• Rethinking Inference Placement for Deep Learning across Edge and Cloud Platforms: A Multi-Objective Optimization Perspective and Future Directions
• Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?
• OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
• Performance Trade-offs of Optimizing Small Language Models for E-Commerce
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
• HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission
• CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation
• Energy-Efficient and Dequantization-Free Q-LLMs: A Spiking Neural Network Approach to Salient Value Mitigation
• ELUTQ: Efficient LUT-Aware Quantization for Deploying Large Language Models on Edge Devices
• EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval
• From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
• EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs
• Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
• Mixed-Precision Quantization for Language Models: Techniques and Prospects
• MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
• FraQAT: Quantization Aware Training with Fractional bits
• AMS-QUANT: Adaptive Mantissa Sharing for Floating-point Quantization
• MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving
• BitNet Distillation
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• MC#: Mixture Compressor for Mixture-of-Experts Large Models
• Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
• PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
• Accelerating Attention with Basis Decomposition
• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
• The AetherFloat Family: Block-Scale-Free Quad-Radix Floating-Point Architectures for AI Accelerators
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models
• Artificial Hippocampus Networks for Efficient Long-Context Modeling
• Training Dynamics Impact Post-Training Quantization Robustness
• lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• QuantDemoire: Quantization with Outlier Aware for Image Demoiréing
• FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networks
• TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAs
• To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
• Post-Training Quantization for Audio Diffusion Transformers
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
• Collaborative Compression for Large-Scale MoE Deployment on Edge
• Layer-wise dynamic rank for compressing large language models
• Pretraining Large Language Models with NVFP4
• CLQ: Cross-Layer Guided Orthogonal-based Quantization for Diffusion Transformers
• Tequila: Trapping-free Ternary Quantization for Large Language Models
• LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
• Beyond Outliers: A Study of Optimizers Under Quantization
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
• RHYTHM: Reasoning with Hierarchical Temporal Tokenization for Human Mobility
• PT²-LLM: Post-Training Ternarization for Large Language Models
• SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights
• InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
• Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
• Compute-Optimal Quantization-Aware Training
• Quantized Visual Geometry Grounded Transformer
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
• Embodied AI: From LLMs to World Models
• LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference
• GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference
• Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution
• QuantWAMs: Calibrating at the Right Granularity for World Action Models
• Linguistic properties and model scale in brain encoding: from small to compressed language models
• Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering
• Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
• On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
• Fair-GPTQ: Bias-Aware Quantization for Large Language Models
• TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
• FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
• Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
• I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation
• ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms
• Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
• Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia
• QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception
• MoPEQ: Mixture of Mixed Precision Quantized Experts
• Dynamic Sparse Attention on Mobile SoCs
• GPT-OSS-20B: A Comprehensive Deployment-Centric Analysis of OpenAI's Open-Weight Mixture of Experts Model
• TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
• Metis: Training LLMs with FP4 Quantization
• Beacon: Post-Training Quantization with Integrated Grid Selection
• APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
• Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistance
• How Quantization Shapes Bias in Large Language Models
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
• Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining
• EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models
• READER: Retrieval-Assisted Drafter for Efficient LLM Inference
• OverFill: Two-Stage Models for Efficient Language Model Decoding
• Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
• Can Smaller Large Language Models Evaluate Research Quality?
• LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimization
• Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
• MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
• DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
• Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos
• Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation
• VLMQ: Efficient Post-Training Quantization for Large Vision-Language Models via Hessian Augmentation
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
• xDeepServe: Model-as-a-Service on Huawei CloudMatrix384
• MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
• EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
• XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
• Unveiling Super Experts in Mixture-of-Experts Large Language Models
• KLLM: Fast LLM Inference with K-Means Quantization
• MSQ: Memory-Efficient Bit Sparsification Quantization

Discussions

• god why did i not know about this. "hurr durr CLIP and diffusion models are hard to train in bf16 because of the high dynamic range of the activations." okay just normalize them before you start train [bsky, 20 points, 5 comments]

Related

• SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM
• AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models
• FPTQ: Fine-grained Post-Training Quantization for Large Language Models
• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
• Post Training Quantization of Large Language Models with Microscaling Formats