PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

2021/08/04 by Machel Reid, Mikel Artetxe, Reid, Machel +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2108.01887

openalex publication_date 2021/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora, and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.

Cited by

• Pretraining Strategies using Monolingual and Parallel Data for Low-Resource Machine Translation

Related

• Multilingual Denoising Pre-training for Neural Machine Translation
• Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models
• On the Role of Parallel Data in Cross-lingual Transfer Learning
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data