Compression Scaling Laws:Unifying Sparsity and Quantization

2025/02/23 by Elias Frantar, Utku Evci, Frantar, Elias +6 · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.16440

openalex publication_date 2025/02/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that weight sparsity acts as a constant multiplier on model size in scaling laws, we demonstrate that this "effective parameter" scaling pattern extends to quantization as well. Specifically, we establish that weight-only quantization achieves strong parameter efficiency multipliers, while full quantization of both weights and activations shows diminishing returns at lower bitwidths. Our results suggest that different compression techniques can be unified under a common scaling law framework, enabling principled comparison and combination of these methods.

Cited by

• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats
• Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
• From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
• Unified Scaling Laws for Compressed Representations
• Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding
• Quartet: Native FP4 Training Can Be Optimal for Large Language Models
• Scaling Law for Quantization-Aware Training
• LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts

Related

• Unified Scaling Laws for Compressed Representations
• From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction
• Compression Laws for Large Language Models
• Scaling Laws for Post Training Quantized Large Language Models
• When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models