Matmul or No Matmul in the Era of 1-bit LLMs

2024/08/21 by Jinendra Malekar, Malekar, Jinendra, Mohammed E. Elbtity +3
Computer Science · Engineering · #Advancements in Semiconductor Devices and Circuit Design #Artificial Intelligence (cs.AI) #Cellular Automata and Applications #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2408.11939

openalex publication_date 2024/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The advent of 1-bit large language models (LLMs) has attracted considerable attention and opened up new research opportunities. However, 1-bit LLMs only improve a fraction of models by applying extreme quantization to the projection layers while leaving attention heads unchanged. Therefore, to avoid fundamentally wrong choices of goals in future research, it is crucial to understand the actual improvements in computation and memory usage that 1-bit LLMs can deliver. In this work, we present an adaptation of Amdahl's Law tailored for the 1-bit LLM context, which illustrates how partial improvements in 1-bit LLMs impact overall model performance. Through extensive experiments, we uncover key nuances across different model architectures and hardware configurations, offering a roadmap for future research in the era of 1-bit LLMs.

Related

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• OneBit: Towards Extremely Low-bit Large Language Models
• PIM-LLM: A High-Throughput Hybrid PIM Architecture for 1-bit LLMs
• Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
• LLM Inference Unveiled: Survey and Roofline Model Insights