Optimizing Transformer for Low-Resource Neural Machine Translation

2020/11/04 by Ali Araabi, Christof Monz, Araabi, Ali +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2011.02266

To be published in COLING 2020

arxiv created 2020/11/04 · arxiv updated 2020/11/05

Abstract

Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.

Related

• On Optimal Transformer Depth for Low-Resource Language Translation
• Enriching the Transformer with Linguistic Factors for Low-Resource Machine Translation
• Heterogeneous Encoders Scaling In The Transformer For Neural Machine Translation
• Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation
• Very Deep Transformers for Neural Machine Translation