Strategies for Training Large Vocabulary Neural Language Models

2016/01/01 by Wenlin Chen, David Grangier, Michael Auli · 103 citations
Computer Science · #Artificial intelligence #Artificial neural network #Computer science #Deep neural networks #Estimator #Language model #Linguistics #Machine learning #Natural Language Processing Techniques #Natural language processing #Normalization (sociology) #Popularity #Softmax function #Speech Recognition and Synthesis #Speech recognition #Statistics #Topic Modeling #Vocabulary

paper · pdf · doi:10.18653/v1/p16-1186

openalex publication_date 2016/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Training neural network language models over large vocabularies is computationally costly compared to count-based models such as Kneser-Ney. We present a systematic comparison of neural strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization. We extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax. We evaluate each method on three popular benchmarks, examining performance on rare words, the speed/accuracy trade-off and complementarity to Kneser-Ney.

Citations

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• Glove: Global Vectors for Word Representation
• Exploring the Limits of Language Modeling

Cited by

• Real-time Neural-based Input Method
• Fast Locality Sensitive Hashing for Beam Search on GPU
• Language Detection Engine for Multilingual Texting on Mobile Devices
• Large-Scale Training System for 100-Million Classification at Alibaba
• Handling Massive N -Gram Datasets Efficiently
• Neural information retrieval: at the end of the early years
• Softmax function [wikipedia]

Related

• Strategies for Training Large Vocabulary Neural Language Models
• Self-organized Hierarchical Softmax
• On Sampling-Based Training Criteria for Neural Language Modeling
• Self-Normalized Importance Sampling for Neural Language Modeling
• Improving Language Modelling with Noise-contrastive estimation