Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages

2022/12/11 by Sumanth Doddapaneni, Rahul Aralikatte, Doddapaneni, Sumanth +10 · 18 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.05409

openalex publication_date 2022/12/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Building Natural Language Understanding (NLU) capabilities for Indic languages, which have a collective speaker base of more than one billion speakers is absolutely crucial. In this work, we aim to improve the NLU capabilities of Indic languages by making contributions along 3 important axes (i) monolingual corpora (ii) NLU testsets (iii) multilingual LLMs focusing on Indic languages. Specifically, we curate the largest monolingual corpora, IndicCorp, with 20.9B tokens covering 24 languages from 4 language families - a 2.3x increase over prior work, while supporting 12 additional languages. Next, we create a human-supervised benchmark, IndicXTREME, consisting of nine diverse NLU tasks covering 20 languages. Across languages and tasks, IndicXTREME contains a total of 105 evaluation sets, of which 52 are new contributions to the literature. To the best of our knowledge, this is the first effort towards creating a standard benchmark for Indic languages that aims to test the multilingual zero-shot capabilities of pretrained language models. Finally, we train IndicBERT v2, a state-of-the-art model supporting all the languages. Averaged across languages and tasks, the model achieves an absolute improvement of 2 points over a strong baseline. The data and models are available at https://github.com/AI4Bharat/IndicBERT.

Cited by

• AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus
• ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages
• IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages
• BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
• destroR: Attacking Transfer Models with Obfuscous Examples to Discard Perplexity
• Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
• IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
• Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus
• Adapting Small Language Models to Low-Resource Domains: A Case Study in Hindi Tourism QA
• Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
• MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages
• DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture
• Quantifying Language Disparities in Multilingual Large Language Models
• Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges
• ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects
• Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
• TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu

Related

• IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
• IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
• MILU: A Multi-task Indic Language Understanding Benchmark
• Paramanu: A Family of Novel Efficient Generative Foundation Language Models for Indian Languages
• Analysis of Indic Language Capabilities in LLMs