Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

2025/10/01 by Eileen Pan, Pan, Eileen, Anna Seo Gyeong Choi +7 · 2 voices
Computer Science · Social Sciences · #Artificial Intelligence in Law #Semantic Web and Ontologies #Statistical and Computational Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2510.00962

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.

Citations

• A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
• Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
• AI generates covertly racist decisions about people based on their dialect
• Careless Whisper: Speech-to-Text Hallucination Harms

Discussions

• It was fantastic to collaborate across Cornell and Apple for our EMNLP paper auditing LLMs for dialectal biases in multiple choice benchmark datasets: arxiv.org/abs/2510.00962. Anna @annaseogyeongchoi [bsky, 17 points, 0 comments]
• 🧵Excited to present our work at #EMNLP2025 “Analyzing Dialectal Biases in LLMs for Knowledge and Reasoning Benchmarks”! Paper 📄 arxiv.org/abs/2510.00962 w/ Eileen Pan, Skyler Seto, @allisonkoe.bsky. [bsky, 7 points, 1 comments]

Related

• Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English
• Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
• EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models
• Language Models can Evaluate Themselves via Probability Discrepancy
• MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs