Depth Growing for Neural Machine Translation

2019/07/03 by Lijun Wu, Yiren Wang, Yi-Ren Wang +15
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1907.01968

Accepted by ACL 2019

arxiv created 2019/07/03 · openalex publication_date 2019/07/03 · arxiv updated 2019/07/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even reduces performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 English→German and English→French translation tasks\footnoteOur code is available at \urlhttps://github.com/apeterswu/Depth_Growing_NMT.

Citations

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation

Related

• Shallow-to-Deep Training for Neural Machine Translation
• Deep Architectures for Neural Machine Translation
• Training Deeper Neural Machine Translation Models with Transparent Attention
• DTMT: A Novel Deep Transition Architecture for Neural Machine Translation
• Very Deep Transformers for Neural Machine Translation