The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages

2025/07/24 by Abdullah, Abdulhady Abas, Gandomi, Amir H., Rashid, Tarik A +4
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.18762

Abstract

In natural language processing, multilingual models like mBERT and XLM-RoBERTa promise broad coverage but often struggle with languages that share a script yet differ in orthographic norms and cultural context. This issue is especially notable in Arabic-script languages such as Kurdish Sorani, Arabic, Persian, and Urdu. We introduce the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language. By focusing pre-training on language-specific script features and statistics, our models capture patterns overlooked by general-purpose models. When fine-tuned on classification tasks, AS-RoBERTa variants outperform mBERT and XLM-RoBERTa by 2 to 5 percentage points. An ablation study confirms that script-focused pre-training is central to these gains. Error analysis using confusion matrices shows how shared script traits and domain-specific content affect performance. Our results highlight the value of script-aware specialization for languages using the Arabic script and support further work on pre-training strategies rooted in script and language specificity.

Citations

• mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences
• BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
• Few-shot Learning with Multilingual Language Models
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Hierarchical Text Classification of Urdu News using Deep Neural Network
• LoRA: Low-Rank Adaptation of Large Language Models
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Beyond English-Centric Multilingual Machine Translation
• Are All Languages Created Equal in Multilingual BERT?
• AraBERT: Transformer-based Model for Arabic Language Understanding
• Unsupervised Cross-lingual Representation Learning at Scale
• Parameter-Efficient Transfer Learning for NLP

Related

• Large Language Models and Arabic Content: A Review
• Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Space
• PALI: A Language Identification Benchmark for Perso-Arabic Scripts
• Sadeed: Advancing Arabic Diacritization Through Small Language Model
• Supporting Undotted Arabic with Pre-trained Language Models