2015/10/11 by Zhouhan Lin, Lin, Zhouhan, Matthieu Courbariaux +5 · 2 voices · 155 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Algorithm #Arithmetic #Artificial intelligence #Artificial neural network #Backpropagation #Binary number #Computation #Computer science #Deep neural networks #Gradient descent #Layer (electronics) #MNIST database #Mathematics #Multiplication (music) #Neural Networks and Applications #Pattern recognition (psychology) #Point (geometry) #Sign (mathematics) #Stochastic Gradient Optimization Techniques #Stochastic gradient descent #Training (meteorology) #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.1510.03009
published in arXiv (Cornell University) (Cornell University) · Published as a conference paper at ICLR 2016. 9 pages, 3 figures
openalex publication_date 2015/10/11 · arxiv created 2016/02/26 · arxiv updated 2016/02/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.