Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis

2025/08/27 by Kamath, Anusha, Singla, Kanishk, Paul, Rakesh +4
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.19831

Abstract

Evaluating instruction-tuned Large Language Models (LLMs) in Hindi is challenging due to a lack of high-quality benchmarks, as direct translation of English datasets fails to capture crucial linguistic and cultural nuances. To address this, we introduce a suite of five Hindi LLM evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi. These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process. We leverage this suite to conduct an extensive benchmarking of open-source LLMs supporting Hindi, providing a detailed comparative analysis of their current capabilities. Our curation process also serves as a replicable methodology for developing benchmarks in other low-resource languages.

Citations

• IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages
• IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
• Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
• MILU: A Multi-task Indic Language Understanding Benchmark
• Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus
• L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context
• INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
• Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
• Instruction-Following Evaluation for Large Language Models
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• GPT-4 Technical Report
• Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• On the Dangers of Stochastic Parrots
• Measuring Massive Multitask Language Understanding
• HellaSwag: Can a Machine Really Finish Your Sentence?

Related

• Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi
• Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi
• Analysis of Indic Language Capabilities in LLMs
• Airavata: Introducing Hindi Instruction-tuned LLM
• IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages