2020/01/31 by Antonio Carta, Alessandro Sperduti, Carta, Antonio +3 · 2 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #Neural Networks and Applications #cs.LG #cs.NE #stat.ML
paper · pdf · doi:10.48550/arxiv.2001.11771
preprint submitted at Elsevier Neural Networks
arxiv created 2020/01/31 · arxiv updated 2020/02/03
Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. We extend the memorization component with a modular memory that encodes the hidden state sequence at different sampling frequencies. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. The experimental results on synthetic and real-world datasets show that specializing the training algorithm to train the memorization component always improves the final performance whenever the memorization of long sequences is necessary to solve the problem.