2021/11/12 by Maxat Tezekbayev, Tezekbayev, Maxat, Vassilina Nikoulina +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2111.06832
openalex publication_date 2021/11/12 · openalex created_date 2022/11/06 · openalex updated_date 2026/07/28
Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits. However, by producing a dense probability distribution each token in the vocabulary has a nonzero chance of being selected at each generation step, leading to a variety of reported problems in text generation. α-entmax of Peters et al. (2019, arXiv:1905.05702) solves this problem, but is considerably slower than softmax. In this paper, we propose an alternative to α-entmax, which keeps its virtuous characteristics, but is as fast as optimized softmax and achieves on par or better performance in machine translation task.