Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators

2024/01/25 by Yaniv Blumenfeld, Blumenfeld, Yaniv, Itay Hubara +3 · 1 citation
Computer Science · Mathematics · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.2401.14110

openalex publication_date 2024/01/25 · openalex created_date 2024/01/27 · openalex updated_date 2026/07/28

Abstract

The majority of the research on the quantization of Deep Neural Networks (DNNs) is focused on reducing the precision of tensors visible by high-level frameworks (e.g., weights, activations, and gradients). However, current hardware still relies on high-accuracy core operations. Most significant is the operation of accumulating products. This high-precision accumulation operation is gradually becoming the main computational bottleneck. This is because, so far, the usage of low-precision accumulators led to a significant degradation in performance. In this work, we present a simple method to train and fine-tune high-end DNNs, to allow, for the first time, utilization of cheaper, 12-bits accumulators, with no significant degradation in accuracy. Lastly, we show that as we decrease the accumulation precision further, using fine-grained gradient approximations can improve the DNN accuracy.

Cited by

• PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training

Related

• TensorQuant - A Simulation Toolbox for Deep Neural Network Quantization
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Mixed Precision DNNs: All you need is a good parametrization
• Value-aware Quantization for Training and Inference of Neural Networks
• Low-bit Quantization of Neural Networks for Efficient Inference