Training deep neural networks with low precision multiplications

2014/12/22 by Matthieu Courbariaux, Yoshua Bengio, Courbariaux, Matthieu +3 · 13 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Numerical Methods and Algorithms

paper · doi:10.48550/arxiv.1412.7024

openalex publication_date 2014/12/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training. We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications.

Citations

• Improving neural networks by preventing co-adaptation of feature detectors
• Gradient-based learning applied to document recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers
• Resource-Efficient Neural Networks for Embedded Systems
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for\n Scientific Data
• Efficient Non-linear Calculators
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Computational Cost Reduction in Learned Transform Classifications
• What Do Compressed Deep Neural Networks Forget?
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• Creating Robust Deep Neural Networks With Coded Distributed Computing for IoT Systems
• LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition
• An Energy-Efficient FPGA-Based Deconvolutional Neural Networks Accelerator for Single Image Super-Resolution
• A CNN Accelerator on FPGA Using Depthwise Separable Convolution

Related

• Deep Neural Network Training without Multiplications
• Deep Learning with Limited Numerical Precision
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
• Low-Precision Floating-Point Schemes for Neural Network Training