QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

2024/02/06 by Albert Tseng, Jerry Chee, Tseng, Albert +8 · 1 voice · 118 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cryptography and Data Security #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2402.04396

openalex publication_date 2024/02/06 · arxiv published 2024/02/06 · arxiv updated 2024/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that achieves state-of-the-art results in extreme compression regimes (≤ 4 bits per weight) using three novel techniques. First, QuIP# improves QuIP's (Chee et al., 2023) incoherence processing by using the randomized Hadamard transform, which is faster and has better theoretical properties. Second, QuIP# uses vector quantization to take advantage of the ball-shaped sub-Gaussian distribution that incoherent weights possess: specifically, we introduce a set of hardware-efficient codebooks based on the highly symmetric E₈ lattice, which achieves the optimal 8-dimension unit ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original model. Our experiments show that QuIP# outperforms existing PTQ methods, enables new behaviors in PTQ scaling, and supports fast inference. Our code can be found at https://github.com/Cornell-RelaxML/quip-sharp.

Cited by

• A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference
• NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
• LLA: Enhancing Security and Privacy for Generative Models with Logic-Locked Accelerators
• GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models
• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs
• SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skipping
• From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models
• SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
• ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers
• Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in \± 1, ± i\
• LPCD: Unified Framework from Layer-Wise to Submodule Quantization
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
• CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
• Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
• Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
• PocketLLM: Ultimate Compression of Large Language Models via Meta Networks
• ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference
• SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantization
• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
• Block Rotation is All You Need for MXFP4 Quantization
• DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
• QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
• ELUTQ: Efficient LUT-Aware Quantization for Deploying Large Language Models on Edge Devices
• MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Network
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• Neural Weight Compression for Language Models
• MC#: Mixture Compressor for Mixture-of-Experts Large Models
• Training Dynamics Impact Post-Training Quantization Robustness
• Collaborative Compression for Large-Scale MoE Deployment on Edge
• Pretraining Large Language Models with NVFP4
• Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
• Low-bit Model Quantization for Deep Neural Networks: A Survey
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
• Effective Quantization of Muon Optimizer States
• PT²-LLM: Post-Training Ternarization for Large Language Models
• SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights
• Quantized Visual Geometry Grounded Transformer
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
• RSAVQ: Riemannian Sensitivity-Aware Vector Quantization for Large Language Models
• LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference
• GyRot: Leveraging Hidden Synergy Between Rotation and Fine-Grained Group Quantization for Low-Bit LLM Inference
• Elucidating the Design Space of FP4 training
• PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models
• SBVR: Summation of BitVector Representation for Efficient LLM Quantization
• Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs
• ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms
• SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
• Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective
• Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos
• FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design
• Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
• CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
• The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
• SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
• Adaptive Block-Scaled Data Types
• A Comprehensive Evaluation on Quantization Techniques for Large Language Models
• On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization
• CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs
• Orthogonal Finetuning Made Scalable
• Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
• DBellQuant: Breaking the Bell with Double-Bell Transformation for LLMs Post Training Binarization
• MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models
• ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Models
• AQUATIC-Diff: Additive Quantization for Truly Tiny Compressed Diffusion Models
• PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling
• FPTQuant: Function-Preserving Transforms for LLM Quantization
• SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
• Unifying Uniform and Binary-coding Quantization for Accurate Compression of Large Language Models
• Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
• QuantFace: Efficient Quantization for Face Restoration
• DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
• Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design
• Highly Efficient and Effective LLMs with Multi-Boolean Architectures
• Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
• Efficient Large Language Model Inference with Neural Block Linearization
• BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models
• BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
• Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding
• DVD-Quant: Data-free Video Diffusion Transformers Quantization
• NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs
• NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
• LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation
• FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
• Hadamard-Domain Model Quantization for Learned Image Coding
• Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
• Optimal Formats for Weight Quantisation
• Quantum Knowledge Distillation for Large Language Models
• High-Rate Nested-Lattice Quantized Matrix Multiplication with Small Lookup Tables
• One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models
• Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization
• Addition is almost all you need: Compressing large language models with double binary factorization
• QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads
• Turning LLM Activations Quantization-Friendly
• GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
• Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free
• TurboESM: Ultra-Efficient 3-Bit KV Cache Quantization for Protein Language Models with Orthogonal Rotation and QJL Correction
• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
• Price of metric universality in vector quantization is at most 0.11 bit
• ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
• ICQuant: Index Coding enables Low-bit LLM Quantization
• GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
• Separating Intelligence from Inference: A Standard for Edge-Native AI Computing
• Theory-optimal Quantization Based on Flatness
• When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
• Dissecting Quantization Error: A Concentration-Alignment Perspective
• A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU
• Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
• M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
• Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference
• NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models
• Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation
• One Qubit Can Beat One Bit: Quantum Advantage for Post-Training Quantization
• ML For Hardware Design Interpretability: Challenges and Opportunities

Discussions

• QuIP#: Even Better LLM Quantization with Hadamard Incoherence, Lattice Codebooks [hn, 2 points, 0 comments]

Related

• QTIP: Quantization with Trellises and Incoherence Processing
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression