Efficient softmax approximation for GPUs

2016/09/14 by Édouard Grave, Edouard Grave, Grave, Edouard +9 · 14 citations
Computer Science · #Advanced Neural Network Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Parallel Computing and Optimization Techniques #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1609.04309

Accepted to ICML 2017

openalex publication_date 2016/09/14 · arxiv created 2017/06/19 · arxiv updated 2017/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computation time. Our approach further reduces the computational time by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax. The code of our method is available at https://github.com/facebookresearch/adaptive-softmax.

Citations

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Finding Structure in Time
• Learning Longer Memory in Recurrent Neural Networks
• Estimation of probabilities from sparse data for the language model component of a speech recognizer
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Exploring the Limits of Language Modeling
• Learning Visual Features from Large Weakly Supervised Data

Cited by

• Rectified SpaAttn: Revisiting Attention Sparsity for Efficient Video Generation
• CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
• Multimodal Regression for Enzyme Turnover Rates Prediction
• A Survey on Non-Intrusive ASR Refinement: From Output-Level Correction to Full-Model Distillation
• v-PuNNs: van der Put Neural Networks for Transparent Ultrametric Representation Learning
• SpeLLM: Character-Level Multi-Head Decoding
• VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits
• Compact Recurrent Transformer with Persistent Memory
• Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
• Integrating Discrete and Neural Features via Mixed-feature Trans-dimensional Random Field Language Models
• Lightweight Adaptive Mixture of Neural and N-gram Language Models
• Concept Mining via Embedding
• Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora
• Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling

Related

• Maximizing Parallelism in Distributed Training for Huge Neural Networks
• Self-organized Hierarchical Softmax
• Exploring the power of GPU's for training Polyglot language models
• Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model