2021/09/13 by Alexander Jones, William Yang Wang, Jones, Alex +4
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2109.06324
openalex publication_date 2021/09/13 · openalex created_date 2022/10/01 · openalex updated_date 2026/07/28
In cross-lingual language models, representations for many different\nlanguages live in the same space. Here, we investigate the linguistic and\nnon-linguistic factors affecting sentence-level alignment in cross-lingual\npretrained language models for 101 languages and 5,050 language pairs. Using\nBERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our\ncorpus, we compute a task-based measure of cross-lingual alignment in the form\nof bitext retrieval performance, as well as four intrinsic measures of vector\nspace alignment and isomorphism. We then examine a range of linguistic,\nquasi-linguistic, and training-related features as potential predictors of\nthese alignment metrics. The results of our analyses show that word order\nagreement and agreement in morphological complexity are two of the strongest\nlinguistic predictors of cross-linguality. We also note in-family training data\nas a stronger predictor than language-specific training data across the board.\nWe verify some of our linguistic findings by looking at the effect of\nmorphological segmentation on English-Inuktitut alignment, in addition to\nexamining the effect of word order agreement on isomorphism for 66 zero-shot\nlanguage pairs from a different corpus. We make the data and code for our\nexperiments publicly available.\n