Quantization Networks

2019/11/21 by Yang, Jiwei, Shen, Xu, Xing, Jun +5 · 12 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1911.09464

Abstract

Although deep neural networks are highly effective, their high computational and memory costs severely challenge their applications on portable devices. As a consequence, low-bit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research topic. Existing methods formulate the low-bit quantization of networks as an approximation or optimization problem. Approximation-based methods confront the gradient mismatch problem, while optimization-based methods are only suitable for quantizing weights and could introduce high computational cost in the training stage. In this paper, we propose a novel perspective of interpreting and implementing neural network quantization by formulating low-bit quantization as a differentiable non-linear function (termed quantization function). The proposed quantization function can be learned in a lossless and end-to-end manner and works for any weights and activations of neural networks in a simple and uniform way. Extensive experiments on image classification and object detection tasks show that our quantization networks outperform the state-of-the-art methods. We believe that the proposed method will shed new insights on the interpretation of neural network quantization. Our code is available at https://github.com/aliyun/alibabacloud-quantization-networks.

Cited by

• AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
• DualHash: A Stochastic Primal-Dual Algorithm with Theoretical Guarantee for Deep Hashing
• Quantized Skeletal Learning (QSL): A Differentiable Programming Approach for Skeletal Reduction of Chemical Mechanisms
• Binary Autoencoder for Mechanistic Interpretability of Large Language Models
• Deep Lookup Network
• Delta-SVD: Efficient Compression for Personalized Text-to-Image Models
• T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
• Data Exfiltration by Compression Attack: Definition and Evaluation on Medical Image Data
• TopKD: Top-scaled Knowledge Distillation
• Test-Time Model Adaptation for Quantized Neural Networks
• MSQ: Memory-Efficient Bit Sparsification Quantization

Related

• Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
• Searching for Low-Bit Weights in Quantized Neural Networks
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Differentiable, Bit-shifting, and Scalable Quantization without training neural network from scratch