Quantized Neural Network Inference with Precision Batching

2020/02/26 by Lam, Maximilian, Yedidia, Zachary, Banbury, Colby +1
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Performance (cs.PF)

paper · doi:10.48550/arxiv.2003.00822

Abstract

We present PrecisionBatching, a quantized inference algorithm for speeding up neural network execution on traditional hardware platforms at low bitwidths without the need for retraining or recalibration. PrecisionBatching decomposes a neural network into individual bitlayers and accumulates them using fast 1-bit operations while maintaining activations in full precision. PrecisionBatching not only facilitates quantized inference at low bitwidths (< 8 bits) without the need for retraining/recalibration, but also 1) enables traditional hardware platforms the ability to realize inference speedups at a finer granularity of quantization (e.g: 1-16 bit execution) and 2) allows accuracy and speedup tradeoffs at runtime by exposing the number of bitlayers to accumulate as a tunable parameter. Across a variety of applications (MNIST, language modeling, natural language inference) and neural network architectures (fully connected, RNN, LSTM), PrecisionBatching yields end-to-end speedups of over 8x on a GPU within a < 1% error margin of the full precision baseline, outperforming traditional 8-bit quantized inference by over 1.5x-2x at the same error tolerance.

Related

• Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance
• A White Paper on Neural Network Quantization
• Low-bit Quantization of Neural Networks for Efficient Inference