Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks

2019/03/03 by Bokang Zhu, Richong Zhang, Zhu, Bokang +5
Computer Science · Mathematics · Psychology · #Artificial intelligence #Artificial neural network #Cognitive psychology #Computer science #FOS: Computer and information sciences #Feature (linguistics) #Feature selection #Linguistics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Memorization #Neural Networks and Applications #Pattern recognition (psychology) #Philosophy #Psychology #Selection (genetic algorithm) #Time Series Analysis and Forecasting #Topic Modeling #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1903.00906

published in arXiv (Cornell University) (Cornell University)

arxiv created 2019/03/03 · openalex publication_date 2019/03/03 · arxiv updated 2019/03/05 · openalex created_date 2019/03/11 · openalex updated_date 2026/07/28

Abstract

In this paper, we propose a test, called Flagged-1-Bit (F1B) test, to study the intrinsic capability of recurrent neural networks in sequence learning. Four different recurrent network models are studied both analytically and experimentally using this test. Our results suggest that in general there exists a conflict between feature selection and feature memorization in sequence learning. Such a conflict can be resolved either using a gating mechanism as in LSTM, or by increasing the state dimension as in Vanilla RNN. Gated models resolve this conflict by adaptively adjusting their state-update equations, whereas Vanilla RNN resolves this conflict by assigning different dimensions different tasks. Insights into feature selection and memorization in recurrent networks are given.

Citations

• On the difficulty of training Recurrent Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• Finding Structure in Time
• Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
• Neural Machine Translation in Linear Time
• Full-Capacity Unitary Recurrent Neural Networks
• DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
• Opening the Black Box of Deep Neural Networks via Information
• Deep learning

Related

• Prototypical Recurrent Unit
• Recurrent neural networks: vanishing and exploding gradients are not the end of the story
• Assessing the Memory Ability of Recurrent Neural Networks
• Encoding-based Memory Modules for Recurrent Neural Networks
• A Critical Review of Recurrent Neural Networks for Sequence Learning