WRPN: Wide Reduced-Precision Networks

2017/09/04 by Asit Mishra, Mishra, Asit, Eriko Nurvitadhi +6 · 4 citations
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE) #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1709.01134

arxiv created 2017/09/04 · openalex publication_date 2017/09/04 · arxiv updated 2017/09/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

Citations

• Deep Learning with Limited Numerical Precision
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Resiliency of Deep Neural Networks under Quantization
• Wide Residual Networks
• Ternary Weight Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Trained Ternary Quantization
• Deep Residual Learning for Image Recognition

Cited by

• BiKAN: Restoring Collapsed Basis of Binary Kolmogorov--Arnold Networks
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation

Related

• WRPN: Training and Inference using Wide Reduced-Precision Networks
• Low Precision RNNs: Quantizing RNNs Without Losing Accuracy
• Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets
• Progressive Learning of Low-Precision Networks
• Low Precision Neural Networks using Subband Decomposition