Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense

2024/05/07 by Siqi Shen, Shen, Siqi, Lajanugen Logeswaran +9 · 1 voice · 13 citations
Computer Science · #Natural Language Processing Techniques #cs.CL

paper · pdf · doi:10.48550/arxiv.2405.04655

Abstract

Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs' general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks. Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally aware language models.

Cited by

• Do Large Language Models Truly Understand Cross-cultural Differences?
• Artwork Interpretation with Vision Language Models: A Case Study on Emotions and Emotion Symbols
• Large Language Models Develop Novel Social Biases Through Adaptive Exploration
• Scaling Cultural Resources for Improving Generative Models
• Quality-Aware Translation Tagging in Multilingual RAG system
• Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs
• The Curious Case of Curiosity across Human Cultures and LLMs
• Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset
• Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning
• DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World
• Everyday Physics in Korean Contexts: A Culturally Grounded Physical Reasoning Benchmark
• Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki

Discussions

• Capabilities and Limitations of Large Language Models for Cultural Commonsense [hn, 2 points, 0 comments]

Related

• Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset
• Entangled in Representations: Mechanistic Investigation of Cultural Biases in Large Language Models
• Global PIQA: Evaluating Physical Commonsense Reasoning Across 100+ Languages and Cultures
• MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs
• Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology