Efficient softmax approximation for GPUs

2016/09/14 by Édouard Grave, Armand Joulin, Grave, Edouard +7 · 6 citations
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1609.04309

openalex publication_date 2016/09/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.

Citations

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Finding Structure in Time
• Learning Longer Memory in Recurrent Neural Networks
• Estimation of probabilities from sparse data for the language model component of a speech recognizer
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Exploring the Limits of Language Modeling
• Learning Visual Features from Large Weakly Supervised Data

Cited by

• Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation
• CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
• Multimodal Regression for Enzyme Turnover Rates Prediction
• A Survey on Non-Intrusive ASR Refinement: From Output-Level Correction to Full-Model Distillation
• v-PuNNs: van der Put Neural Networks for Transparent Ultrametric Representation Learning
• SpeLLM: Character-Level Multi-Head Decoding

Related

• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• Self-organized Hierarchical Softmax
• Exploring the power of GPU's for training Polyglot language models
• Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model