On the Compression of Recurrent Neural Networks with an Application to\n LVCSR acoustic modeling for Embedded Speech Recognition

2016/03/25 by Rohit Prabhavalkar, Ouais Alsharif, Prabhavalkar, Rohit +5 · 13 citations
Computer Science · #Acoustic model #Artificial intelligence #Artificial neural network #Compression (physics) #Computer science #Data compression #Focus (optics) #Layer (electronics) #Long short term memory #Music and Audio Processing #Neural Networks and Applications #Pattern recognition (psychology) #Recurrent neural network #Speech Recognition and Synthesis #Speech and Audio Processing #Speech processing #Speech recognition #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1603.08042

published in arXiv (Cornell University) (Cornell University) · Accepted in ICASSP 2016

openalex publication_date 2016/03/25 · arxiv created 2016/05/02 · arxiv updated 2016/05/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04

Abstract

We study the problem of compressing recurrent neural networks (RNNs). In\nparticular, we focus on the compression of RNN acoustic models, which are\nmotivated by the goal of building compact and accurate speech recognition\nsystems which can be run efficiently on mobile devices. In this work, we\npresent a technique for general recurrent model compression that jointly\ncompresses both recurrent and non-recurrent inter-layer weight matrices. We\nfind that the proposed technique allows us to reduce the size of our Long\nShort-Term Memory (LSTM) acoustic model to a third of its original size with\nnegligible loss in accuracy.\n

Citations

• Predicting Parameters in Deep Learning

Cited by

• The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning
• Sequence-Level Knowledge Distillation
• Masked multi-head self-attention for causal speech enhancement
• Deep Learning in Mobile and Wireless Networking: A Survey
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
• An Embedded Deep Learning based Word Prediction
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency
• Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• In-training Matrix Factorization for Parameter-frugal Neural Machine Translation
• Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices

Related

• Learning Compact Recurrent Neural Networks
• Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition
• Parameter Compression of Recurrent Neural Networks and Degradation of Short-term Memory
• High Order Recurrent Neural Networks for Acoustic Modelling
• Frame Stacking and Retaining for Recurrent Neural Network Acoustic Model