On the Compression of Recurrent Neural Networks with an Application to\n LVCSR acoustic modeling for Embedded Speech Recognition

2016/03/25 by Rohit Prabhavalkar, Ouais Alsharif, Prabhavalkar, Rohit +5 · 7 citations
Computer Science · #Speech Recognition and Synthesis #Speech and Audio Processing #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1603.08042

Abstract

We study the problem of compressing recurrent neural networks (RNNs). In\nparticular, we focus on the compression of RNN acoustic models, which are\nmotivated by the goal of building compact and accurate speech recognition\nsystems which can be run efficiently on mobile devices. In this work, we\npresent a technique for general recurrent model compression that jointly\ncompresses both recurrent and non-recurrent inter-layer weight matrices. We\nfind that the proposed technique allows us to reduce the size of our Long\nShort-Term Memory (LSTM) acoustic model to a third of its original size with\nnegligible loss in accuracy.\n

Citations

• Predicting Parameters in Deep Learning

Cited by

• The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning
• Sequence-Level Knowledge Distillation
• Masked multi-head self-attention for causal speech enhancement
• Deep Learning in Mobile and Wireless Networking: A Survey
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
• An Embedded Deep Learning based Word Prediction
• Learning Intrinsic Sparse Structures within Long Short-Term Memory

Related

• Learning Compact Recurrent Neural Networks
• Fast and Accurate Recurrent Neural Network Acoustic Models for Speech\n Recognition
• Parameter Compression of Recurrent Neural Networks and Degradation of\n Short-term Memory
• High Order Recurrent Neural Networks for Acoustic Modelling
• Frame Stacking and Retaining for Recurrent Neural Network Acoustic Model