PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian

2025/02/11 by Monazzah, Erfan Moosavi, Rahimzadeh, Vahid, Yaghoobzadeh, Yadollah +2
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #I.2.7

paper · doi:10.48550/arxiv.2502.07459

Abstract

Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here: https://huggingface.co/datasets/teias-ai/percul

Related

• MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language
• FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models
• Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning
• PBBQ: A Persian Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models
• Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?