Pushing the Limits of Large Language Model Quantization via the Linearity Theorem

2024/11/26 by Vladimir Malinovskii, Andrei Panferov, Malinovskii, Vladimir +10 · 6 voices · 5 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2411.17525

openalex publication_date 2024/11/26 · openalex created_date 2024/12/05 · openalex updated_date 2026/07/28

Abstract

Quantizing large language models has become a standard way to reduce their memory and computational costs. Typically, existing methods focus on breaking down the problem into individual layer-wise sub-problems, and minimizing per-layer error, measured via various metrics. Yet, this approach currently lacks theoretical justification and the metrics employed may be sub-optimal. In this paper, we present a "linearity theorem" establishing a direct relationship between the layer-wise ℓ₂ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, which outperforms all prior data-free approaches such as the extremely popular NF4 quantized format, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels which match a given compression constraint in the medium-bitwidth regime, obtained by reduction to dynamic programming. On the practical side, we demonstrate improved accuracy-compression trade-offs on Llama-3.1 and 3.2-family models, as well as on Qwen-family models. Further, we show that our method can be efficiently supported in terms of GPU kernels at various batch sizes, advancing both data-free and non-uniform quantization for LLMs.

Cited by

• Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
• Pretraining Large Language Models with NVFP4
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
• Binary Quantization For LLMs Through Dynamic Grouping

Discussions

• Pushing the Limits of LLM Quantization via the Linearity Theorem [hn, 95 points, 2 comments]
• Pushing the Limits of LLM Quantization via the Linearity Theorem https://arxiv.org/abs/2411.17525 [comments] [40 points] [bsky, 1 points, 0 comments]
• Pushing the Limits of LLM Quantization via the Linearity Theorem https://arxiv.org/abs/2411.17525 (https://news.ycombinator.com/item?id=43744343) [bsky, 0 points, 0 comments]
• Pushing the Limits of LLM Quantization via the Linearity Theorem https://arxiv.org/abs/2411.17525 (https://news.ycombinator.com/item?id=43744343) [bsky, 0 points, 0 comments]
• Pushing the Limits of LLM Quantization via the Linearity Theorem [bsky, 0 points, 0 comments]
• Pushing the Limits of LLM Quantization via the Linearity Theorem https://arxiv.org/abs/2411.17525 [bsky, 0 points, 0 comments]

Related

• QuantEase: Optimization-based Quantization for Language Models
• EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
• A Speed Odyssey for Deployable Quantization of LLMs
• CBQ: Cross-Block Quantization for Large Language Models
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit