Mixed Precision DNNs: All you need is a good parametrization

2019/05/27 by Stefan Uhlich, Lukas Mauch, Uhlich, Stefan +13 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1905.11452

openalex publication_date 2019/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.

Citations

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Ternary Weight Networks
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• A Survey of FPGA-Based Neural Network Accelerator
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Confounding Tradeoffs for Neural Network Quantization

Related

• FracBits: Mixed Precision Quantization via Fractional Bit-Widths
• Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks
• AdaQAT: Adaptive Bit-Width Quantization-Aware Training
• BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch
• Mixed-Precision Neural Networks: A Survey