It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT

2020/10/16 by Gonen, Hila, Ravfogel, Shauli, Elazar, Yanai +1 · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2010.08275

Abstract

Recent works have demonstrated that multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages. We study the word-level translation information embedded in mBERT and present two simple methods that expose remarkable translation capabilities with no fine-tuning. The results suggest that most of this information is encoded in a non-linear way, while some of it can also be recovered with purely linear tools. As part of our analysis, we test the hypothesis that mBERT learns representations which contain both a language-encoding component and an abstract, cross-lingual component, and explicitly identify an empirical language-identity subspace within mBERT representations.

Cited by

• InstructNet: A Novel Approach for Multi-Label Instruction Classification through Advanced Deep Learning
• MaiBERT: A Pre-training Corpus and Language Model for Low-Resourced Maithili Language

Related

• How Language-Neutral is Multilingual BERT?
• Identifying Necessary Elements for BERT's Multilinguality
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT
• How multilingual is Multilingual BERT?
• Finding Universal Grammatical Relations in Multilingual BERT