LQER: Low-Rank Quantization Error Reconstruction for LLMs

2024/02/04 by Zhang, Cheng, Cheng, Jianyi, Constantinides, George A. +1 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2402.02446

Abstract

Post-training quantization of Large Language Models (LLMs) is challenging. In this work, we introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. LQER leverages an activation-induced scale matrix to drive the singular value distribution of quantization error towards a desirable distribution, which enables nearly-lossless W4A8 quantization on various LLMs and downstream tasks without the need for knowledge distillation, grid search, or gradient-base iterative optimization. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes to collect high-precision weights from irregular memory locations. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36× fewer hardware resources than the leading state-of-the-art method. We open-source our framework at https://github.com/ChengZhang-98/lqer

Cited by

• MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
• Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation

Related

• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization
• ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
• Low-Rank Correction for Quantized LLMs
• QERA: an Analytical Framework for Quantization Error Reconstruction
• LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices