An Analysis of Neural Language Modeling at Multiple Scales

2018/03/22 by Stephen Merity, Merity, Stephen, Nitish Shirish Keskar +3 · 28 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.1803.08240

Abstract

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.

Cited by

• Addressing Some Limitations of Transformers with Feedback Memory
• Evaluating Computational Language Models with Scaling Properties of Natural Language
• Calibration, Entropy Rates, and Memory in Language Models
• Revisiting Simple Neural Probabilistic Language Models
• Neural Polysynthetic Language Modelling
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Analysing Dropout and Compounding Errors in Neural Language Models
• EG-MLA: Embedding-Gated Multi-head Latent Attention for Scalable and Efficient LLMs
• A Survey of Deep Learning for Scientific Discovery
• GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors
• Single Headed Attention RNN: Stop Thinking With Your Head
• Progress and Tradeoffs in Neural Language Models
• Trellis Networks for Sequence Modeling
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• A Tensorized Transformer for Language Modeling
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Character-based Surprisal as a Model of Reading Difficulty in the\n Presence of Error
• Contextual Text Embeddings for Twi
• On Extending NLP Techniques from the Categorical to the Latent Space: KL Divergence, Zipf's Law, and Similarity Search
• The Implicit and Explicit Regularization Effects of Dropout
• Exploring Self-Identified Counseling Expertise in Online Support Forums
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• DeLighT: Deep and Light-weight Transformer
• E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks
• Discrete Flows: Invertible Generative Models of Discrete Data
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• Learning Architectures from an Extended Search Space for Language Modeling
• Character n-gram Embeddings to Improve RNN Language Models

Related

• Character-Aware Neural Language Models
• Exploring the Limits of Language Modeling
• Scaling Recurrent Neural Network Language Models
• On the State of the Art of Evaluation in Neural Language Models
• Syntax-Aware Language Modeling with Recurrent Neural Networks