Quantization Networks

2019/11/21 by Jiwei Yang, Yang, Jiwei, Xu Shen +13 · 26 citations
Computer Science · Mathematics · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1911.09464

10 pages, CVPR2019

arxiv created 2019/11/28 · arxiv updated 2019/12/02

Abstract

Although deep neural networks are highly effective, their high computational and memory costs severely challenge their applications on portable devices. As a consequence, low-bit quantization, which converts a full-precision neural network into a low-bitwidth integer version, has been an active and promising research topic. Existing methods formulate the low-bit quantization of networks as an approximation or optimization problem. Approximation-based methods confront the gradient mismatch problem, while optimization-based methods are only suitable for quantizing weights and could introduce high computational cost in the training stage. In this paper, we propose a novel perspective of interpreting and implementing neural network quantization by formulating low-bit quantization as a differentiable non-linear function (termed quantization function). The proposed quantization function can be learned in a lossless and end-to-end manner and works for any weights and activations of neural networks in a simple and uniform way. Extensive experiments on image classification and object detection tasks show that our quantization networks outperform the state-of-the-art methods. We believe that the proposed method will shed new insights on the interpretation of neural network quantization. Our code is available at https://github.com/aliyun/alibabacloud-quantization-networks.

Cited by

• AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Models
• DualHash: A Stochastic Primal-Dual Algorithm with Theoretical Guarantee for Deep Hashing
• Quantized Skeletal Learning (QSL): A Differentiable Programming Approach for Skeletal Reduction of Chemical Mechanisms
• Binary Autoencoder for Mechanistic Interpretability of Large Language Models
• Deep Lookup Network
• Delta-SVD: Efficient Compression for Personalized Text-to-Image Models
• T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
• Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications
• Data Exfiltration by Compression Attack: Definition and Evaluation on Medical Image Data
• CompLeak: Deep Learning Model Compression Exacerbates Privacy Leakage
• TopKD: Top-scaled Knowledge Distillation
• Test-Time Model Adaptation for Quantized Neural Networks
• MSQ: Memory-Efficient Bit Sparsification Quantization
• Edge Intelligence with Spiking Neural Networks
• BitWave: Exploiting Column-Based Bit-Level Sparsity for Deep Learning Acceleration
• Red grape detection with accelerated artificial neural networks in the FPGA's programmable logic
• Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
• TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision
• ReStNet: A Reusable & Stitchable Network for Dynamic Adaptation on IoT Devices
• Diffusion Model Quantization: A Review
• Learning Binarized Representations with Pseudo-positive Sample Enhancement for Efficient Graph Collaborative Filtering
• Energy Considerations for Large Pretrained Neural Networks
• Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance
• Shifting AI Efficiency From Model-Centric to Data-Centric Compression
• Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning
• DNAD: Differentiable Neural Architecture Distillation

Related

• Quantization and Training of Low Bit-Width Convolutional Neural Networks for Object Detection
• Searching for Low-Bit Weights in Quantized Neural Networks
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks
• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Differentiable, Bit-shifting, and Scalable Quantization without training neural network from scratch