LCQ: Low-Rank Codebook based Quantization for Large Language Models

2024/05/31 by Wen-Pu Cai, Cai, Wen-Pu, Wujun Li +2 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2405.20973

openalex publication_date 2024/05/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models~(LLMs) have recently demonstrated promising performance in many tasks. However, the high storage and computational cost of LLMs has become a challenge for deploying LLMs. Weight quantization has been widely used for model compression, which can reduce both storage and computational cost. Most existing weight quantization methods for LLMs use a rank-one codebook for quantization, which results in substantial accuracy loss when the compression ratio is high. In this paper, we propose a novel weight quantization method, called low-rank codebook based quantization~(LCQ), for LLMs. LCQ adopts a low-rank codebook, the rank of which can be larger than one, for quantization. Experiments show that LCQ can achieve better accuracy than existing methods with a negligibly extra storage cost.

Cited by

• Neural Weight Compression for Language Models

Related

• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
• Foundations of Large Language Model Compression -- Part 1: Weight Quantization
• SLMQuant:Benchmarking Small Language Model Quantization for Practical Deployment
• CDQuant: Greedy Coordinate Descent for Accurate LLM Quantization
• LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices