GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

2025/05/11 by Jinuk Kim, Marwa El Halabi, Kim, Jinuk +12 · 7 citations
Computer Science · #End-to-end principle #Inference #Language model #Monotonic function #Natural Language Processing Techniques #Quantization (signal processing) #Source code #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.07004

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/11 · openalex created_date 2025/10/16 · openalex updated_date 2026/08/05

Abstract

Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.

Citations

• Log-Time K-Means Clustering for 1D Data: Novel Approaches with Proof and Implementation
• SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
• CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
• QTIP: Quantization with Trellises and Incoherence Processing
• SpinQuant: LLM quantization with learned rotations
• PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• AI and Memory Wall
• GPTVQ: The Blessing of Dimensionality for LLM Quantization
• Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Extreme Compression of Large Language Models via Additive Quantization
• Fairness-Aware Structured Pruning in Transformers
• QuantEase: Optimization-based Quantization for Language Models
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• SqueezeLLM: Dense-and-Sparse Quantization
• QLoRA: Efficient Finetuning of Quantized LLMs
• LLaMA: Open and Efficient Foundation Language Models
• The case for 4-bit precision: k-bit Inference Scaling Laws
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
• The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
• 8-bit Optimizers via Block-wise Quantization
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Measuring Massive Multitask Language Understanding
• Up or Down? Adaptive Rounding for Post-Training Quantization
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• Importance Estimation for Neural Network Pruning
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Faster gaze prediction with dense networks and Fisher pruning
• Fast Exact k-Means, k-Medians and Bregman Divergence Clustering in 1D
• Towards the Limit of Network Quantization
• Pointer Sentinel Mixture Models
• Deep Residual Learning for Image Recognition
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Least squares quantization in PCM

Cited by

• NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
• P/D-Device: Disaggregated Large Language Model between Cloud and Devices
• A Comprehensive Evaluation on Quantization Techniques for Large Language Models
• Discrete and Continuous Difference of Submodular Minimization
• Model-Preserving Adaptive Rounding

Related

• SiLQ: Simple Large Language Model Quantization-Aware Training
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners
• CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
• LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid
• OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting