T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

2024/06/25 by Jianyu Wei, Shijie Cao, Ting Cao +4 · 2 voices · 2 citations
Computer Science · Engineering · #Advanced Data Storage Technologies #Advanced Memory and Neural Computing #Parallel Computing and Optimization Techniques #cs.AI #cs.DC

paper · pdf · doi:10.1145/3689031.3696099

arxiv published 2024/06/25 · arxiv updated 2025/03/25 · openalex publication_date 2025/03/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead.

Citations

• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• High Throughput Matrix-Matrix Multiplication between Asymmetric Bit-Width Operands
• LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• The case for 4-bit precision: k-bit Inference Scaling Laws
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

Cited by

• BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
• LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation

Discussions

• T-Mac: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge [hn, 2 points, 0 comments]
• arxiv.org/pdf/2407.00088 [bsky, 0 points, 0 comments]