Why Do Some Inputs Break Low-Bit LLM Quantization?

2025/05/24 by Ting-Yun Chang, Chang, Ting-Yun, Zhang, Muru +4 · 1 citation
Engineering · #Advancements in Photolithography Techniques #Advancements in Semiconductor Devices and Circuit Design #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2506.12044

openalex publication_date 2025/05/24 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Low-bit weight-only quantization significantly reduces the memory footprint of large language models (LLMs), but disproportionately affects certain examples. We analyze diverse 3-4 bit methods on LLMs ranging from 7B-70B in size and find that the quantization errors of 50 pairs of methods are strongly correlated (avg. 0.82) on FineWeb examples. Moreover, the residual stream magnitudes of full-precision models are indicative of future quantization errors. We further establish a hypothesis that relates the residual stream magnitudes to error amplification and accumulation over layers. Using LLM localization techniques, early exiting, and activation patching, we show that examples with large errors rely on precise residual activations in the late layers, and that the outputs of MLP gates play a crucial role in maintaining the perplexity. Our work reveals why certain examples result in large quantization errors and which model components are most critical for performance preservation.

Cited by

• LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

Related

• Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
• Turning LLM Activations Quantization-Friendly
• What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
• DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization
• Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens