Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

2025/04/08 by Peerat Limkonchotiwat, Limkonchotiwat, Peerat, Kanruethai Masuk +11 · 1 citation
Computer Science · #Natural Language Processing Techniques #Authorship Attribution and Profiling #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.05898

Abstract

Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency

Cited by

• JAI-1: A Thai-Centric Large Language Model

Related

• Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models
• Typhoon: Thai Large Language Models
• WangchanThaiInstruct: An instruction-following Dataset for Culture-Aware, Multitask, and Multi-domain Evaluation in Thai
• SeaLLMs -- Large Language Models for Southeast Asia
• BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models