Learning low-precision neural networks without Straight-Through Estimator(STE)

2019/03/04 by Zhigang Liu, Liu, Zhi-Gang, Matthew Mattina +1 · 1 citation
Computer Science · Medicine · #Advanced Neural Network Applications #COVID-19 diagnosis using AI #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1903.01061

openalex publication_date 2019/03/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low-precision using stochastic gradient descent (SGD). Our method (AB) avoids STE approximation by replacing the quantized weight in the loss function by an affine combination of the quantized weight w_q and the corresponding full-precision weight w with non-trainable scalar coefficient α and 1-α. During training, α is gradually increased from 0 to 1; the gradient updates to the weights are through the full-precision term, (1-α)w, of the affine combination; the model is converted from full-precision to low-precision progressively. To evaluate the method, a 1-bit BinaryNet on CIFAR10 dataset and 8-bits, 4-bits MobileNet v1, ResNet₅0 v1/2 on ImageNet dataset are trained using the alpha-blending approach, and the evaluation indicates that AB improves top-1 accuracy by 0.9%, 0.82% and 2.93% respectively compared to the results of STE based quantization.

Citations

• Binarized Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Cited by

• Training Deep Normalization-Free Spiking Neural Networks with Lateral Inhibition

Related

• Training Quantized Neural Networks with a Full-precision Auxiliary Module
• Progressive Element-wise Gradient Estimation for Neural Network Quantization
• Improving the Straight-Through Estimator with Zeroth-Order Information
• Network Quantization with Element-wise Gradient Scaling
• Robust Training of Neural Networks at Arbitrary Precision and Sparsity