2021/02/01 by Nora Kassner, Philipp Dufter, Kassner, Nora +3 · 13 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2102.00894
openalex publication_date 2021/02/01 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Recently, it has been found that monolingual English language models can be\nused as knowledge bases. Instead of structural knowledge base queries, masked\nsentences such as "Paris is the capital of [MASK]" are used as probes. We\ntranslate the established benchmarks TREx and GoogleRE into 53 languages.\nWorking with mBERT, we investigate three questions. (i) Can mBERT be used as a\nmultilingual knowledge base? Most prior work only considers English. Extending\nresearch to multiple languages is important for diversity and accessibility.\n(ii) Is mBERT's performance as knowledge base language-independent or does it\nvary from language to language? (iii) A multilingual model is trained on more\ntext, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for\nbetter performance? We find that using mBERT as a knowledge base yields varying\nperformance across languages and pooling predictions across languages improves\nperformance. Conversely, mBERT exhibits a language bias; e.g., when queried in\nItalian, it tends to predict Italy as the country of origin.\n