AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for\n Indic Languages

2020/04/30 by Anoop Kunchukuttan, Divyanshu Kakwani, Kunchukuttan, Anoop +12 · 5 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2005.00085

Abstract

We present the IndicNLP corpus, a large-scale, general-domain corpus\ncontaining 2.7 billion words for 10 Indian languages from two language\nfamilies. We share pre-trained word embeddings trained on these corpora. We\ncreate news article category classification datasets for 9 languages to\nevaluate the embeddings. We show that the IndicNLP embeddings significantly\noutperform publicly available pre-trained embedding on multiple evaluation\ntasks. We hope that the availability of the corpus will accelerate Indic NLP\nresearch. The resources are available at\nhttps://github.com/ai4bharat-indicnlp/indicnlp_corpus.\n

Cited by

• BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
• BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languages
• BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages
• MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment

Related

• Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages
• L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages
• Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
• Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languages
• "A Passage to India": Pre-trained Word Embeddings for Indian Languages