Loss-aware Weight Quantization of Deep Networks

2018/02/23 by Lu Hou, Hou, Lu, James T. Kwok +1 · 1 citation
Computer Science · Engineering · Neuroscience · #Advanced Neural Network Applications #Brain Tumor Detection and Classification #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sparse and Compressive Sensing Techniques

paper · pdf · doi:10.48550/arxiv.1802.08635

openalex publication_date 2018/02/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The huge size of deep networks hinders their use in small computing devices. In this paper, we consider compressing the network by weight quantization. We extend a recently proposed loss-aware weight binarization scheme to ternarization, with possibly different scaling parameters for the positive and negative weights, and m-bit (where m > 2) quantization. Experiments on feedforward and recurrent neural networks show that the proposed scheme outperforms state-of-the-art weight quantization algorithms, and is as accurate (or even more accurate) than the full-precision network.

Citations

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Long Short-Term Memory
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Ternary Weight Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Trained Ternary Quantization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Training Quantized Nets: A Deeper Understanding
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications

Cited by

• Radio: Rate-Distortion Optimization for Large Language Model Compression

Related

• Adaptive Loss-aware Quantization for Multi-bit Networks
• Weight Fixing Networks
• Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization
• Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization
• Model compression as constrained optimization, with application to neural nets. Part II: quantization