Kuwain 1.5B: An Arabic SLM via Language Injection

2025/04/21 by Hennara, Khalil, Chrouf, Sara, Hamed, Mohamed Motaism +3 · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2504.15120

Abstract

Enhancing existing models with new knowledge is a crucial aspect of AI development. This paper introduces a novel method for integrating a new language into a large language model (LLM). Our approach successfully incorporates a previously unseen target language into an existing LLM without compromising its prior knowledge. We trained a tiny model with 1.5 billion parameters named Kuwain by injecting the Arabic language into a small open-source model mainly trained in English. Our method demonstrates significant improvements in Arabic language performance, with an average 8% improvement across various benchmarks, while retaining the model's existing knowledge with a minimum amount of the original model's data. This offers a cost-effective alternative to training a comprehensive model in both English and Arabic. The results highlight the potential for efficient, targeted language model expansion without extensive retraining or resource-intensive processes.

Cited by

• Progressive Depth Up-scaling via Optimal Transport
• Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Model

Related

• CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks
• ALLaM: Large Language Models for Arabic and English
• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
• Large Language Models and Arabic Content: A Review
• Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM