Radio: Rate-Distortion Optimization for Large Language Model Compression

2025/05/05 by Sean I. Young, Young, Sean I. · 1 citation
Computer Science · #Advanced Data Compression Techniques

paper · pdf · doi:10.48550/arxiv.2505.03031

Abstract

In recent years, the compression of large language models (LLMs) has emerged as a key problem in facilitating LLM deployment on resource-limited devices, reducing compute costs, and mitigating the environmental footprint due to large-scale AI infrastructure. Here, we establish the foundations of LLM quantization from a rate-distortion theory perspective and propose a quantization technique based on simple rate-distortion optimization. Our technique scales to models containing hundreds of billions of weight parameters and offers users the flexibility to compress models, post-training, to a model size or accuracy specified by the user.

Citations

• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• LLM Inference Unveiled: Survey and Roofline Model Insights
• GPTVQ: The Blessing of Dimensionality for LLM Quantization
• Extreme Compression of Large Language Models via Additive Quantization
• Compressing LLMs: The Truth is Rarely Pure and Never Simple
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees
• SqueezeLLM: Dense-and-Sparse Quantization
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Adaptive Data-Free Quantization
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• OPT: Open Pre-trained Transformer Language Models
• Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
• Training Verifiers to Solve Math Word Problems
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization
• Generative Low-bitwidth Data Free Quantization
• Adaptive Loss-aware Quantization for Multi-bit Networks
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Loss Aware Post-training Quantization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Learned Step Size Quantization
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• Loss-aware Weight Quantization of Deep Networks
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Pointer Sentinel Mixture Models
• Compressing Deep Convolutional Networks using Vector Quantization
• Least squares quantization in PCM

Cited by

• Optimal Formats for Weight Quantisation

Related

• Foundations of Large Language Model Compression -- Part 1: Weight Quantization
• Aggressive Post-Training Compression on Extremely Large Language Models
• When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models
• Compression Laws for Large Language Models
• BackSlash: Rate Constrained Optimized Training of Large Language Models