Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translation

2023/06/02 by Shravan Nayak, Surangika Ranathunga, Nayak, Shravan +15
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2306.01382

Abstract

NMT systems trained on Pre-trained Multilingual Sequence-Sequence (PMSS) models flounder when sufficient amounts of parallel data is not available for fine-tuning. This specifically holds for languages missing/under-represented in these models. The problem gets aggravated when the data comes from different domains. In this paper, we show that intermediate-task fine-tuning (ITFT) of PMSS models is extremely beneficial for domain-specific NMT, especially when target domain data is limited/unavailable and the considered languages are missing or under-represented in the PMSS model. We quantify the domain-specific results variations using a domain-divergence test, and show that ITFT can mitigate the impact of domain divergence to some extent.

Related

• Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation
• Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?
• Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation
• Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translation
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data