Model compression as constrained optimization, with application to neural nets. Part II: quantization

2017/07/13 by Miguel Á. Carreira-Perpiñán, Carreira-Perpiñán, Miguel Á., Yerlan Idelbayev +1 · 5 citations
Computer Science · #Advanced Data Compression Techniques #FOS: Computer and information sciences #FOS: Mathematics #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.1707.04319

openalex publication_date 2017/07/13 · openalex created_date 2017/07/21 · openalex updated_date 2026/07/28

Abstract

We consider the problem of deep neural net compression by quantization: given a large, reference net, we want to quantize its real-valued weights using a codebook with K entries so that the training loss of the quantized net is minimal. The codebook can be optimally learned jointly with the net, or fixed, as for binarization or ternarization approaches. Previous work has quantized the weights of the reference net, or incorporated rounding operations in the backpropagation algorithm, but this has no guarantee of converging to a loss-optimal, quantized net. We describe a new approach based on the recently proposed framework of model compression as constrained optimization \citepCarreir17a. This results in a simple iterative "learning-compression" algorithm, which alternates a step that learns a net of continuous weights with a step that quantizes (or binarizes/ternarizes) the weights, and is guaranteed to converge to local optimum of the loss for quantized nets. We develop algorithms for an adaptive codebook or a (partially) fixed codebook. The latter includes binarization, ternarization, powers-of-two and other important particular cases. We show experimentally that we can achieve much higher compression rates than previous quantization work (even using just 1 bit per weight) with negligible loss degradation.

Citations

• Compressing Deep Convolutional Networks using Vector Quantization
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Gradient-based learning applied to document recognition
• Ternary Weight Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Predicting Parameters in Deep Learning

Cited by

• A Discrepancy-Based Perspective on Dataset Condensation
• Joint Quantization and Pruning Neural Networks Approach: A Case Study on FSO Receivers
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks

Related

• Model compression as constrained optimization, with application to neural nets. Part V: combining compressions
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Training with Quantization Noise for Extreme Model Compression
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning