Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference

2026/08/04 by Junyi Luo, Xinting Jiang, Tai-Hao Wen +9
Computer Science · #cs.AR

paper · pdf

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix either the element format or the precision-recovery scheme across blocks, and thus capture only limited quantization heterogeneity. Quantization heterogeneity appears at two levels: 1) across blocks, the preferred element format and precision-recovery scheme vary; 2) across operands, weights and activations require different encoding. We introduce AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator. It selects the precision-recovery scheme per block and the representation per operand, at no increase in equivalent bit width (EBW). One design covers two block sizes, giving a higher-accuracy operating point and a lower-EBW operating point that saves storage. We implement a 22nm FD-SOI AI accelerator prototype with the proposed decoder, computing unit, and quantization logic. Against an otherwise identical MXFP4 accelerator with FP4-only multipliers, AdaMX adds about 1% system energy. At the lower-EBW point, AdaMX stays more accurate than the baseline while lowering both memory footprint and energy. Across LLMs from 3B to 70B, AdaMX removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss. AdaMX also generalizes to multimodal models. On Gemma-4 12B, it leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.

Citations

• OpenAI GPT-5 System Card
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving
• Recipes for Pre-training LLMs with MXFP8
• Efficient Precision-Scalable Hardware for Microscaling (MX) Processing in Robotics Learning
• SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training
• BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models
• SCALE-Sim v3: A modular cycle-accurate systolic accelerator simulator for end-to-end system analysis
• M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type
• BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference
• Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models
• MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantization
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Extreme Compression of Large Language Models via Additive Quantization
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
• Microscaling Data Formats for Deep Learning
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• SqueezeLLM: Dense-and-Sparse Quantization
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• With Shared Microexponents, A Little Shifting Goes a Long Way
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• Measuring Massive Multitask Language Understanding
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• PIQA: Reasoning about Physical Commonsense in Natural Language
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Towards VQA Models That Can Read
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Pointer Sentinel Mixture Models
• A Diagram Is Worth A Dozen Images
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge