Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding

2025/05/24 by Alexander Conzelmann, Conzelmann, Alexander, Robert Bamler +1
Computer Science · Engineering · #CCD and CMOS Imaging Sensors #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural Networks and Reservoir Computing

paper · pdf · doi:10.48550/arxiv.2505.18758

openalex publication_date 2025/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The ever-growing size of neural networks poses serious challenges on resource-constrained devices, such as embedded sensors. Compression algorithms that reduce their size can mitigate these problems, provided that model performance stays close to the original. We propose a novel post-training compression framework that combines rate-aware quantization with entropy coding by (1) extending the well-known layer-wise loss by a quadratic rate estimation, and (2) providing locally exact solutions to this modified objective following the Optimal Brain Surgeon (OBS) method. Our method allows for very fast decoding and is compatible with arbitrary quantization grids. We verify our results empirically by testing on various computer-vision networks, achieving a 20-40% decrease in bit rate at the same performance as the popular compression algorithm NNCodec. Our code is available at https://github.com/Conzel/cerwu.

Citations

• Compression Scaling Laws:Unifying Sparsity and Quantization
• NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
• STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
• Inference Optimization of Foundation Models on AI Accelerators
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
• GPTVQ: The Blessing of Dimensionality for LLM Quantization
• Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
• QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
• Extreme Compression of Large Language Models via Additive Quantization
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Memory Efficient Optimizers with 4-bit States
• A Simple and Effective Pruning Approach for Large Language Models
• SqueezeLLM: Dense-and-Sparse Quantization
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
• QLoRA: Efficient Finetuning of Quantized LLMs
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• The case for 4-bit precision: k-bit Inference Scaling Laws
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• Understanding Entropy Coding With Asymmetric Numeral Systems (ANS): a Statistician's Perspective
• A White Paper on Neural Network Quantization
• Energy-Efficient Model Compression and Splitting for Collaborative Inference Over Time-Varying Channels
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Up or Down? Adaptive Rounding for Post-Training Quantization
• CAT: Compression-Aware Training for bandwidth reduction
• Searching for MobileNetV3
• Compact and Computationally Efficient Representation of Deep Neural Networks
• Universal Deep Neural Network Compression
• Deep Residual Learning for Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Related

• Efficient Neural Compression with Inference-time Decoding
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• Bandwidth-efficient Inference for Neural Image Compression
• Entropy-Constrained Training of Deep Neural Networks
• Rate-Distortion Optimized Post-Training Quantization for Learned Image Compression