Evaluating Cultural Knowledge Processing in Large Language Models: A Cognitive Benchmarking Framework Integrating Retrieval-Augmented Generation

2025/11/03 by Hung-Shin Lee, Lee, Hung-Shin, Chen‐Chi Chang +5
Social Sciences · Arts and Humanities · #Computational and Text Analysis Methods #Digital Humanities and Scholarship #Language and cultural evolution

paper · pdf · doi:10.48550/arxiv.2511.01649

Abstract

This study proposes a cognitive benchmarking framework to evaluate how large language models (LLMs) process and apply culturally specific knowledge. The framework integrates Bloom's Taxonomy with Retrieval-Augmented Generation (RAG) to assess model performance across six hierarchical cognitive domains: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. Using a curated Taiwanese Hakka digital cultural archive as the primary testbed, the evaluation measures LLM-generated responses' semantic accuracy and cultural relevance.

Citations

• Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis
• Assessing AI-Generated Questions' Alignment with Cognitive Frameworks in Educational Assessment
• We're Different, We're the Same: Creative Homogeneity Across LLMs
• Evaluating Creative Short Story Generation in Humans and Large Language Models
• Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives
• COGNET-MD, an evaluation framework and dataset for Large Language Model benchmarks in the medical domain
• A Survey on Retrieval-Augmented Text Generation for Large Language Models
• Retrieval-Augmented Generation for AI-Generated Content: A Survey
• Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling
• Assessing and Understanding Creativity in Large Language Models
• AI Hallucinations: A Misnomer Worth Clarifying
• TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models
• Retrieval-Augmented Generation for Large Language Models: A Survey
• MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• Rethinking the Evaluating Framework for Natural Language Understanding in AI Systems: Language Acquisition as a Core for Future Metrics
• S³: Social-network Simulation System with Large Language Model-Empowered Agents
• A Survey on Evaluation of Large Language Models
• Measuring Massive Multitask Language Understanding
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization

Related

• Benchmarking Cognitive Domains for LLMs: Insights from Taiwanese Hakka Culture
• CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis
• Towards Geo-Culturally Grounded LLM Generations
• FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models
• On Classification with Large Language Models in Cultural Analytics