Improving Neural Machine Translation Models with Monolingual Data
2015/11/20 by Rico Sennrich, Barry Haddow, Sennrich, Rico +3 · 51 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL
paper · pdf · doi:10.48550/arxiv.1511.06709
accepted to ACL 2016; new section on effect of back-translation quality
arxiv created 2016/06/03 · arxiv updated 2016/06/06
Abstract
Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Target-side monolingual data plays an important role in boosting fluency for phrase-based statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with separately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolingual data without changing the neural network architecture. By pairing monolingual training data with an automatic back-translation, we can treat it as additional parallel training data, and we obtain substantial improvements on the WMT 15 task English<->German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish->English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives substantial improvements for the IWSLT 15 task English->German.
Cited by
- Neural Machine Translation for Low-Resource Tangkhul--English
- NagaNLP: Bootstrapping NLP for Low-Resource Nagamese Creole with Human-in-the-Loop Synthetic Data
- THeGAU: Type-Aware Heterogeneous Graph Autoencoder and Augmentation
- Multilingual Medical Reasoning for Question Answering with Large Language Models
- An Interpretability-Guided Framework for Responsible Synthetic Data Generation in Emotional Text
- NSL-MT: Linguistically Informed Negative Samples for Efficient Machine Translation in Low-Resource Languages
- DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation
- Reversal Invariance in Autoregressive Language Models
- MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data
- BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks
- LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
- Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models
- Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation
- Filtering with Confidence: When Data Augmentation Meets Conformal Prediction
- CCQA: Generating Question from Solution Can Improve Inference-Time Reasoning in SLMs
- AISTAT lab system for DCASE2025 Task6: Language-based audio retrieval
- Building Large-Scale English-Romanian Literary Translation Resources with Open Models
- CURE: Controlled Unlearning for Robust Embeddings -- Mitigating Conceptual Shortcuts in Pre-Trained Language Models
- Explicit and Implicit Data Augmentation for Social Event Detection
- Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader
- Abex-rat: Synergizing Abstractive Augmentation and Adversarial Training for Classification of Occupational Accident Reports
- Towards On-Device Personalization: Cloud-device Collaborative Data Augmentation for Efficient On-device Language Model
- CoBA: Counterbias Text Augmentation for Mitigating Various Spurious Correlations via Semantic Triples
- Transplant Then Regenerate: A New Paradigm for Text Data Augmentation
- DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization
- TopXGen: Topic-Diverse Parallel Data Generation for Low-Resource Machine Translation
- Score Before You Speak: Improving Persona Consistency in Dialogue Generation using Response Quality Scores
- A French Version of the OLDI Seed Corpus
- SLoW: Select Low-frequency Words! Automatic Dictionary Selection for Translation on Large Language Models
- Restoring Rhythm: Punctuation Restoration Using Transformer Models for Bangla, a Low-Resource Language
- QiMeng-MuPa: Mutual-Supervised Learning for Sequential-to-Parallel Code Translation
- Cross-Lingual Transfer for Machine Translation in Turkic Languages
- Measuring Intent Comprehension in LLMs
- Is linguistically-motivated data augmentation worth it?
- Explainable AI: XAI-Guided Context-Aware Data Augmentation
- Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation
- GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
- Bemba Speech Translation: Exploring a Low-Resource African Language
- Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs
- From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech Recognition
- The Effects of Data Augmentation on Confidence Estimation for LLMs
- Pivot Language for Low-Resource Machine Translation
- Scaling Low-Resource MT via Synthetic Data Generation with LLMs
- SMOTExT: SMOTE meets Large Language Models
- Synthetic Data RL: Task Definition Is All You Need
- Towards High-Fidelity Synthetic Multi-platform Social Media Datasets via Large Language Models
- Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data
- TabletCraft: Bridging a 4,000-Year Cultural Gap with Bidirectional Akkadian NMT and Cuneiform Rendering
- Batch Aggregation: An Approach to Enhance Text Classification with Correlated Augmented Data
- Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends
- AttentionDrop: A Novel Regularization Method for Transformer Models
- Simulator-Grounded Large Language Models for Industrial Causal Reasoning: Tool-Use, Structured Injection, and Plant-Portable Retrieval for Wastewater Treatment Decision Support
Related