Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps

2025/10/15 by Ahmed Alzubaidi, Shaikha Alsuwaidi, Alzubaidi, Ahmed +13 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2510.13430

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

This survey provides the first systematic review of Arabic LLM benchmarks, analyzing 40+ evaluation benchmarks across NLP tasks, knowledge domains, cultural understanding, and specialized capabilities. We propose a taxonomy organizing benchmarks into four categories: Knowledge, NLP Tasks, Culture and Dialects, and Target-Specific evaluations. Our analysis reveals significant progress in benchmark diversity while identifying critical gaps: limited temporal evaluation, insufficient multi-turn dialogue assessment, and cultural misalignment in translated datasets. We examine three primary approaches: native collection, translation, and synthetic generation discussing their trade-offs regarding authenticity, scale, and cost. This work serves as a comprehensive reference for Arabic NLP researchers, providing insights into benchmark methodologies, reproducibility standards, and evaluation metrics while offering recommendations for future development.

Citations

• AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
• PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture
• MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering
• AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data
• 3LM: Bridging Arabic, STEM, and Code through Benchmarking
• The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology
• Revisiting Common Assumptions about Arabic Dialects in NLP
• Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs
• ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark
• Large Language Models and Arabic Content: A Review
• MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks
• YourBench: Easy Custom Evaluation Sets for Everyone
• HalluVerse25: Fine-grained Multilingual Benchmark Dataset for LLM Hallucinations
• Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs
• Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking
• Commonsense Reasoning in Arab Culture
• AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects
• AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic
• A Survey of Large Language Models for Arabic Language and its Dialects
• CAMEL-Bench: A Comprehensive Arabic LMM Benchmark
• Arabic Dataset for LLM Safeguard Evaluation
• CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks
• AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs
• ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models
• The Qiyas Benchmark: Measuring ChatGPT Mathematical and Language Understanding in Arabic
• AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic
• Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
• AceGPT, Localizing Large Language Models in Arabic
• A Comprehensive Overview of Large Language Models
• Dolphin: A Challenging and Diverse Benchmark for Arabic NLG
• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
• ORCA: A Challenging Benchmark for Arabic Language Understanding
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• AraGPT2: Pre-Trained Transformer for Arabic Language Generation
• EXAMS: A Multi-Subject High School Examinations Dataset for\n Cross-Lingual and Multilingual Question Answering
• Measuring Massive Multitask Language Understanding
• TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
• Neural Arabic Question Answering

Cited by

• AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

Related

• BALSAM: A Platform for Benchmarking Arabic Large Language Models
• Large Language Models and Arabic Content: A Review
• From Guidelines to Practice: A New Paradigm for Arabic Language Model Evaluation
• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM
• ORCA: A Challenging Benchmark for Arabic Language Understanding