ConLID: Supervised Contrastive Learning for Low-Resource Language Identification

2025/06/18 by Foroutan, Negar, Saydaliev, Jakhongir, Kim, Ye Eun +1 · 2 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2506.15304

Abstract

Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource languages -- often limited to single-domain data, such as the Bible -- continue to perform poorly. To resolve these class imbalance and bias issues, we propose a novel supervised contrastive learning (SCL) approach to learn domain-invariant representations for low-resource languages. Through an extensive analysis, we show that our approach improves LID performance on out-of-domain data for low-resource languages by 3.2%, demonstrating its effectiveness in enhancing LID models.

Cited by

• What Language is This? Ask Your Tokenizer
• DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching

Related

• It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMs
• CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
• LIDE: Language Identification from Text Documents
• Low-resource Languages: A Review of Past Work and Future Challenges
• DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching