On Using Monolingual Corpora in Neural Machine Translation

2015/03/11 by Caglar Gulcehre, Çağlar Gülçehre, Gulcehre, Caglar +18 · 515 citations
Computer Science · #Artificial intelligence #Artificial neural network #BLEU #Baseline (sea) #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Leverage (statistics) #Linguistics #Machine translation #Natural Language Processing Techniques #Natural language processing #Phrase #Task (project management) #Text Readability and Simplification #Topic Modeling #Translation (biology) #Turkish #cs.CL

paper · pdf · doi:10.48550/arxiv.1503.03535

published in HAL (Le Centre pour la Communication Scientifique Directe) (Centre National de la Recherche Scientifique) · 9 pages, 2 figures

openalex publication_date 2015/03/11 · arxiv created 2015/06/12 · arxiv updated 2015/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to 1.96 BLEU improvement on the low-resource language pair Turkish-English, and 1.59 BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of 0.39 and 0.47 BLEU scores over the neural machine translation baselines, respectively.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Adam: A Method for Stochastic Optimization
• Theano: new features and speed improvements
• Improving neural networks by preventing co-adaptation of feature detectors
• Long Short-Term Memory
• Bidirectional recurrent neural networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Cited by

• All-in-One ASR: Unifying Encoder-Decoder Models of CTC, Attention, and Transducer in Dual-Mode ASR
• WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Pretrained Speech Recognition Transformers
• A Survey on Non-Intrusive ASR Refinement: From Output-Level Correction to Full-Model Distillation
• FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities
• Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding
• Minimum Risk Training for Neural Machine Translation
• Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscripts
• Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
• LLMs and Speech: Integration vs. Combination
• Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
• Towards the Unseen: Iterative Text Recognition by Distilling from Errors
• Neural Machine Translation with Pivot Languages
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Closing the Indexing-Decoding Gap in Multimodal Generative Retrieval via Prefix Retention Optimization
• Deep Shallow Fusion for RNN-T Personalization

Related

• A Survey of Methods to Leverage Monolingual Data in Low-resource Neural Machine Translation
• Using Monolingual Data in Neural Machine Translation: a Systematic Study
• Phrase-Based & Neural Unsupervised Machine Translation
• Improving Neural Machine Translation Models with Monolingual Data
• Neural machine translation, corpus and frugality