mPMR: A Multilingual Pre-trained Machine Reader at Scale

2023/05/23 by Weiwen Xu, Xu, Weiwen, Xin Li +5
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.13645

openalex publication_date 2023/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present multilingual Pre-trained Machine Reader (mPMR), a novel method for multilingual machine reading comprehension (MRC)-style pre-training. mPMR aims to guide multilingual pre-trained language models (mPLMs) to perform natural language understanding (NLU) including both sequence classification and span extraction in multiple languages. To achieve cross-lingual generalization when only source-language fine-tuning data is available, existing mPLMs solely transfer NLU capability from a source language to target languages. In contrast, mPMR allows the direct inheritance of multilingual NLU capability from the MRC-style pre-training to downstream tasks. Therefore, mPMR acquires better NLU capability for target languages. mPMR also provides a unified solver for tackling cross-lingual span extraction and sequence classification, thereby enabling the extraction of rationales to explain the sentence-pair classification process.

Related

• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension
• From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader
• Empirical study of pretrained multilingual language models for zero-shot cross-lingual knowledge transfer in generation
• A Multilingual Modeling Method for Span-Extraction Reading Comprehension