2019/10/01 by Kenton Murray, Jeffery Kinnison, Murray, Kenton +9 · 1 citation
Computer Science · Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1910.06717
The 3rd Workshop on Neural Generation and Translation (WNGT 2019)
arxiv created 2019/10/01 · openalex publication_date 2019/10/01 · arxiv updated 2019/10/16 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
Neural sequence-to-sequence models, particularly the Transformer, are the state of the art in machine translation. Yet these neural networks are very sensitive to architecture and hyperparameter settings. Optimizing these settings by grid or random search is computationally expensive because it requires many training runs. In this paper, we incorporate architecture search into a single training run through auto-sizing, which uses regularization to delete neurons in a network over the course of training. On very low-resource language pairs, we show that auto-sizing can improve BLEU scores by up to 3.9 points while removing one-third of the parameters from the model.