2019/12/02 by Jeremy W. Gordon, Jeremy Gordon, David Rawlinson +4
Computer Science · Materials Science · Mathematics · #68T05 #68T30 #Artificial intelligence #Artificial neural network #Autoencoder #Boosting (machine learning) #Computation and Language (cs.CL) #Computer science #Deep learning #Embedding #FOS: Computer and information sciences #I.2.4 #I.2.7 #Language model #MNIST database #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Machine learning #Memory footprint #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Recurrent neural network #Sequence learning #Topic Modeling #Transformer #Word embedding #acm:68T05 #acm:68T30 #cs.CL #cs.LG #cs.NE #msc:68T05 #msc:68T30 #stat.ML
paper · pdf · doi:10.48550/arxiv.1912.01116
published in arXiv (Cornell University) (Cornell University) · 9 pages, 6 figures, 4 tables
arxiv created 2019/12/02 · openalex publication_date 2019/12/02 · arxiv updated 2019/12/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/04
In sequence learning tasks such as language modelling, Recurrent Neural\nNetworks must learn relationships between input features separated by time.\nState of the art models such as LSTM and Transformer are trained by\nbackpropagation of losses into prior hidden states and inputs held in memory.\nThis allows gradients to flow from present to past and effectively learn with\nperfect hindsight, but at a significant memory cost. In this paper we show that\nit is possible to train high performance recurrent networks using information\nthat is local in time, and thereby achieve a significantly reduced memory\nfootprint. We describe a predictive autoencoder called bRSM featuring recurrent\nconnections, sparse activations, and a boosting rule for improved cell\nutilization. The architecture demonstrates near optimal performance on a\nnon-deterministic (stochastic) partially-observable sequence learning task\nconsisting of high-Markov-order sequences of MNIST digits. We find that this\nmodel learns these sequences faster and more completely than an LSTM, and offer\nseveral possible explanations why the LSTM architecture might struggle with the\npartially observable sequence structure in this task. We also apply our model\nto a next word prediction task on the Penn Treebank (PTB) dataset. We show that\na 'flattened' RSM network, when paired with a modern semantic word embedding\nand the addition of boosting, achieves 103.5 PPL (a 20-point improvement over\nthe best N-gram models), beating ordinary RNNs trained with BPTT and\napproaching the scores of early LSTM implementations. This work provides\nencouraging evidence that strong results on challenging tasks such as language\nmodelling may be possible using less memory intensive, biologically-plausible\ntraining regimes.\n