Contextual Text Embeddings for Twi

2021/03/29 by Paul Azunre, Salomey Osei, Azunre, Paul +51
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2103.15963

openalex publication_date 2021/03/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer-based language models have been changing the modern Natural Language Processing (NLP) landscape for high-resource languages such as English, Chinese, Russian, etc. However, this technology does not yet exist for any Ghanaian language. In this paper, we introduce the first of such models for Twi or Akan, the most widely spoken Ghanaian language. The specific contribution of this research work is the development of several pretrained transformer language models for the Akuapem and Asante dialects of Twi, paving the way for advances in application areas such as Named Entity Recognition (NER), Neural Machine Translation (NMT), Sentiment Analysis (SA) and Part-of-Speech (POS) tagging. Specifically, we introduce four different flavours of ABENA -- A BERT model Now in Akan that is fine-tuned on a set of Akan corpora, and BAKO - BERT with Akan Knowledge only, which is trained from scratch. We open-source the model through the Hugging Face model hub and demonstrate its use via a simple sentiment classification example.

Citations

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
• Exploring the Limits of Language Modeling
• An Introduction to Convolutional Neural Networks
• An Ensemble Method to Produce High-Quality Word Embeddings (2016)
• Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation
• An Analysis of Neural Language Modeling at Multiple Scales
• LLM-Supported Manufacturing Mapping Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Masakhane -- Machine Translation For Africa

Related

• The Ghanaian NLP Landscape: A First Look
• Development of a WAZOBIA-Named Entity Recognition System
• Harnessing Transfer Learning from Swahili: Advancing Solutions for Comorian Dialects
• NLP for Ghanaian Languages
• AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation of Dialogues in Low-Resource, African Languages