A Survey of Cross-lingual Word Embedding Models

2017/06/30 by Sebastian Ruder, Ivan Vulić, Anders Søgaard · 256 citations
Computer Science · #Embedding #ICT in Developing Communities #Key (lock) #Meaning (existential) #Natural Language Processing Techniques #Theme (computing) #Topic Modeling #Word (group theory) #Word embedding #cs.CL #cs.LG

paper · pdf · doi:10.1613/jair.1.11640

published in Journal of Artificial Intelligence Research 65, 569-631 (AI Access Foundation) · Published in Journal of Artificial Intelligence Research

openalex created_date 2017/12/04 · openalex publication_date 2019/08/12 · arxiv created 2019/10/06 · arxiv updated 2019/10/08 · openalex updated_date 2026/08/05

Abstract

Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages. In this survey, we provide a comprehensive typology of cross-lingual word embedding models. We compare their data requirements and objective functions. The recurring theme of the survey is that many of the models presented in the literature optimize for the same objectives, and that seemingly different models are often equivalent, modulo optimization strategies, hyper-parameters, and such. We also discuss the different ways cross-lingual word embeddings are evaluated, as well as future challenges and research horizons.

Citations

• Learning Multilingual Word Representations using a Bag-of-Words Autoencoder
• SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
• A Generalized Solution of the Orthogonal Procrustes Problem
• Exploiting Similarities among Languages for Machine Translation
• Distributed Representations of Sentences and Documents
• Distributed Representations of Words and Phrases and their Compositionality
• Cross-lingual Models of Word Embeddings: An Empirical Comparison
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Learning Crosslingual Word Embeddings without Bilingual Corpora
• Enriching Word Vectors with Subword Information
• Multilingual Multi-modal Embeddings for Natural Language Processing
• Model Transfer for Tagging Low-resource Languages using a Bilingual Dictionary
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
• Leveraging Monolingual Data for Crosslingual Compositional Word Representations

Cited by

• Tokenization Matters: Improving Zero-Shot NER for Indic Languages
• Creating and Evaluating Code-Mixed Nepali-English and Telugu-English Datasets for Abusive Language Detection Using Traditional and Deep Learning Models