Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition

2017/12/14 by Jinmian Ye, Ye, Jinmian, Linnan Wang +11 · 2 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.1712.05134

openalex publication_date 2017/12/14 · openalex created_date 2017/12/22 · openalex updated_date 2026/07/28

Abstract

Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters. On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6% in the Action Recognition task on the UCF11 dataset.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• DRAW: A Recurrent Neural Network For Image Generation
• Some Mathematical Notes on Three-Mode Factor Analysis
• Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” Decomposition
• Tensor Decompositions and Applications
• Long Short-Term Memory
• Speeding-up Convolutional Neural Networks Using Fine-tuned\n CP-Decomposition
• Learning Compact Recurrent Neural Networks
• Densely Connected Convolutional Networks
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Tensor-Train Recurrent Neural Networks for Video Classification
• Tensorial Recurrent Neural Networks for Longitudinal Data Analysis
• BT-Nets: Simplifying Deep Neural Networks via Block Term Decomposition
• Simple and Efficient Parallelization for Probabilistic Temporal Tensor Factorization
• On the Compression of Recurrent Neural Networks with an Application to\n LVCSR acoustic modeling for Embedded Speech Recognition
• Predicting Parameters in Deep Learning
• Long-term Forecasting using Higher Order Tensor RNNs

Cited by

• Downsized and Compromised?: Assessing the Faithfulness of Model Compression
• A Novel Deep Hybrid Framework with Ensemble-Based Feature Optimization for Robust Real-Time Human Activity Recognition

Related

• Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition
• Tensor-Train Recurrent Neural Networks for Video Classification
• A Fully Tensorized Recurrent Neural Network
• Tensor Decomposition for Compressing Recurrent Neural Network
• Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure