Delta Networks for Optimized Recurrent Network Computation

2016/12/16 by Daniel Neil, Jun Haeng Lee, Neil, Daniel +7 · 42 citations
Computer Science · #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Benchmark (surveying) #Computation #Computer science #Convolutional neural network #FOS: Computer and information sciences #Machine learning #Music and Audio Processing #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Parallel computing #Pattern recognition (psychology) #Recurrent neural network #Speedup #cs.NE

paper · pdf · doi:10.48550/arxiv.1612.05571

published in arXiv (Cornell University), 2584-2593 (Cornell University)

arxiv created 2016/12/16 · openalex publication_date 2016/12/16 · arxiv updated 2016/12/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many neural networks exhibit stability in their activation patterns over time in response to inputs from sensors operating under real-world conditions. By capitalizing on this property of natural signals, we propose a Recurrent Neural Network (RNN) architecture called a delta network in which each neuron transmits its value only when the change in its activation exceeds a threshold. The execution of RNNs as delta networks is attractive because their states must be stored and fetched at every timestep, unlike in convolutional neural networks (CNNs). We show that a naive run-time delta network implementation offers modest improvements on the number of memory accesses and computes, but optimized training techniques confer higher accuracy at higher speedup. With these optimizations, we demonstrate a 9X reduction in cost with negligible loss of accuracy for the TIDIGITS audio digit recognition benchmark. Similarly, on the large Wall Street Journal (WSJ) speech recognition benchmark, pretrained networks can also be greatly accelerated as delta networks and trained delta networks show a 5.7X improvement with negligible loss of accuracy. Finally, on an endto-end CNN-RNN network trained for steering angle prediction in a driving dataset, the RNN cost can be reduced by a substantial 100X.

Citations

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Long Short-Term Memory
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Recurrent Neural Networks With Limited Numerical Precision
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural\n Networks

Cited by

• DeltaLLM: A Training-Free Framework Exploiting Temporal Sparsity for Efficient Edge LLM Inference
• OpenDPDv2: A Unified Learning and Optimization Framework for Neural Network Digital Predistortion
• DeltaDPD: Exploiting Dynamic Temporal Sparsity in Recurrent Neural Networks for Energy-Efficient Wideband Digital Predistortion
• It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware

Related

• Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products
• Training Recurrent Neural Networks against Noisy Computations during Inference
• Gated Delta Networks: Improving Mamba2 with Delta Rule
• iRNN: Integer-only Recurrent Neural Network