Is (Selective) Round-To-Nearest Quantization All You Need?

2025/05/21 by Alex Kogan, Kogan, Alex
Computer Science · #Big Data and Digital Economy #D.4.8 #FOS: Computer and information sciences #G.4 #I.2.7 #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2505.15909

openalex publication_date 2025/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Quantization became a necessary tool for serving ever-increasing Large Language Models (LLMs). RTN (Round-to-Nearest) is perhaps the simplest quantization technique that has been around well before LLMs surged to the forefront of machine learning (ML) research. Yet, it has been largely dismissed by recent and more advanced quantization methods that claim superiority over RTN in nearly every aspect of performance. This work aims to dispel this established point of view, showing that RTN is not only much cheaper to apply, but also its token generation throughput can be better than and accuracy can be similar to more advanced alternatives. In particular, we discuss our implementation of RTN based on the recent Marlin kernels and demonstrate how the accuracy of RTN can be gradually improved by selectively increasing the data precision format of certain model layers and modules. Based on our results, we argue that RTN presents a viable and practical choice for quantizing LLMs.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
• ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• A Survey of Quantization Methods for Efficient Neural Network Inference
• PubMedQA: A Dataset for Biomedical Research Question Answering
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Adaptive Quantization for Deep Neural Network
• Pointer Sentinel Mixture Models
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Related

• Low Precision RNNs: Quantizing RNNs Without Losing Accuracy
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners
• An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
• ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation