From Facts to Folklore: Evaluating Large Language Models on Bengali Cultural Knowledge

2025/10/22 by Chowdhury, Nafis, Haque, Moinul, Ahmed, Anika +4
#Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.20043

Abstract

Recent progress in NLP research has demonstrated remarkable capabilities of large language models (LLMs) across a wide range of tasks. While recent multilingual benchmarks have advanced cultural evaluation for LLMs, critical gaps remain in capturing the nuances of low-resource cultures. Our work addresses these limitations through a Bengali Language Cultural Knowledge (BLanCK) dataset including folk traditions, culinary arts, and regional dialects. Our investigation of several multilingual language models shows that while these models perform well in non-cultural categories, they struggle significantly with cultural knowledge and performance improves substantially across all models when context is provided, emphasizing context-aware architectures and culturally curated training data.

Citations

• Advancements and Challenges in Bangla Question Answering Models: A Comprehensive Review
• Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation
• Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings
• Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs
• ArabicaQA: A Comprehensive Dataset for Arabic Question Answering
• Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models
• Having Beer after Prayer? Measuring Cultural Bias in Large Language Models
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Evaluation of Question Answering Systems: Complexity of judging a natural language
• PaLM: Scaling Language Modeling with Pathways
• A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English

Related

• BLUCK: A Benchmark Dataset for Bengali Linguistic Understanding and Cultural Knowledge
• BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali
• Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs
• Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis
• SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture