Cross-lingual Transfer of Monolingual Models

2021/09/15 by Evangelia Gogoulou, Ariel Ekgren, Gogoulou, Evangelia +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.07348

Accepted to LREC 2022

arxiv created 2022/05/19 · arxiv updated 2022/05/20

Abstract

Recent studies in zero-shot cross-lingual learning using multilingual models have falsified the previous hypothesis that shared vocabulary and joint pre-training are the keys to cross-lingual generalization. Inspired by this advancement, we introduce a cross-lingual transfer method for monolingual models based on domain adaptation. We study the effects of such transfer from four different languages to English. Our experimental results on GLUE show that the transferred models outperform the native English model independently of the source language. After probing the English linguistic knowledge encoded in the representations before and after transfer, we find that semantic information is retained from the source language, while syntactic information is learned during transfer. Additionally, the results of evaluating the transferred models in source language tasks reveal that their performance in the source domain deteriorates after transfer.

Cited by

• Model-Aware Tokenizer Transfer
• Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer

Related

• On the ability of monolingual models to learn language-agnostic representations
• Measuring Cross-lingual Transfer in Bytes
• Cross-Lingual Language Model Meta-Pretraining
• On the Cross-lingual Transferability of Monolingual Representations
• Emerging Cross-lingual Structure in Pretrained Language Models