Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation

2025/10/10 by Muhammad Ali Shafique, Kanwal Mehreen, Shafique, Muhammad Ali +9
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.11 #I.2.6 #I.2.7 #ICT in Developing Communities #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.09051

openalex publication_date 2025/10/10 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

Developing a high-performing large language models (LLMs) for low-resource languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality datasets, multilingual inconsistencies, and safety concerns. Existing multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique prompts and seed values for each task along with a global task pool, this dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under 100. Our results demonstrate that high-performance and low-resource language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach. All datasets, models, and code are publicly available at: https://github.com/traversaal-ai/alif-urdu-llm.

Citations

• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
• The Llama 3 Herd of Models
• Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Language Models are Multilingual Chain-of-Thought Reasoners
• Training Verifiers to Solve Math Word Problems
• LoRA: Low-Rank Adaptation of Large Language Models
• Unsupervised Cross-lingual Representation Learning at Scale
• Massively Multilingual Neural Machine Translation
• Model Cards for Model Reporting

Related

• UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings
• UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu
• Generalists vs. Specialists: Evaluating Large Language Models for Urdu
• UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
• Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks