2013/01/23 by Marius Pachitariu, Maneesh Sahani, Pachitariu, Marius +1 · 4 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1301.5650
openalex publication_date 2013/01/23 · openalex created_date 2022/10/01 · openalex updated_date 2026/07/28
Neural language models (LMs) based on recurrent neural networks (RNN) are\nsome of the most successful word and character-level LMs. Why do they work so\nwell, in particular better than linear neural LMs? Possible explanations are\nthat RNNs have an implicitly better regularization or that RNNs have a higher\ncapacity for storing patterns due to their nonlinearities or both. Here we\nargue for the first explanation in the limit of little training data and the\nsecond explanation for large amounts of text data. We show state-of-the-art\nperformance on the popular and small Penn dataset when RNN LMs are regularized\nwith random dropout. Nonetheless, we show even better performance from a\nsimplified, much less expressive linear RNN model without off-diagonal entries\nin the recurrent matrix. We call this model an impulse-response LM (IRLM).\nUsing random dropout, column normalization and annealed learning rates, IRLMs\ndevelop neurons that keep a memory of up to 50 words in the past and achieve a\nperplexity of 102.5 on the Penn dataset. On two large datasets however, the\nsame regularization methods are unsuccessful for both models and the RNN's\nexpressivity allows it to overtake the IRLM by 10 and 20 percent perplexity,\nrespectively. Despite the perplexity gap, IRLMs still outperform RNNs on the\nMicrosoft Research Sentence Completion (MRSC) task. We develop a slightly\nmodified IRLM that separates long-context units (LCUs) from short-context units\nand show that the LCUs alone achieve a state-of-the-art performance on the MRSC\ntask of 60.8%. Our analysis indicates that a fruitful direction of research for\nneural LMs lies in developing more accessible internal representations, and\nsuggests an optimization regime of very high momentum terms for effectively\ntraining such models.\n