Factorization tricks for LSTM networks

2017/03/31 by Oleksii Kuchaiev, Boris Ginsburg, Kuchaiev, Oleksii +1 · 1 voice · 5 citations
Computer Science · #Neural Networks and Applications #Topic Modeling #Music and Audio Processing

paper · pdf · doi:10.48550/arxiv.1703.10722

Abstract

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.

Citations

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Cited by

• A Lightweight Recurrent Network for Sequence Modeling
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification
• Driving with Data: Modeling and Forecasting Vehicle Fleet Maintenance in Detroit
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN

Discussions

• Factorization tricks for LSTM networks [hn, 69 points, 14 comments]

Related

• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Slim LSTM networks: LSTM₆ and LSTM_C6
• SLIM LSTMs
• Simplified Long Short-term Memory Recurrent Neural Networks: part III
• Benchmarking of LSTM Networks