The Challenge of Diacritics in Yoruba Embeddings

2020/11/15 by Tosin Adewumi, Adewumi, Tosin P., Foteini Liwicki +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2011.07605

openalex publication_date 2020/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The major contributions of this work include the empirical establishment of a better performance for Yoruba embeddings from undiacritized (normalized) dataset and provision of new analogy sets for evaluation. The Yoruba language, being a tonal language, utilizes diacritics (tonal marks) in written form. We show that this affects embedding performance by creating embeddings from exactly the same Wikipedia dataset but with the second one normalized to be undiacritized. We further compare average intrinsic performance with two other work (using analogy test set & WordSim) and we obtain the best performance in WordSim and corresponding Spearman correlation.

Citations

• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word\n Embeddings
• Word2Vec: Optimal Hyper-Parameters and Their Impact on NLP Downstream Tasks

Related

• Improving Yor\`ub'a Diacritic Restoration
• Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yorùbá and Twi
• YAD: Leveraging T5 for Improved Automatic Diacritization of Yorùbá Text
• Bridging Gaps in Natural Language Processing for Yorùbá: A Systematic Review of a Decade of Progress and Prospects
• Attentive Sequence-to-Sequence Learning for Diacritic Restoration of Yorùbá Language Text