XNLI: Evaluating Cross-lingual Sentence Representations
2018/09/13 by Alexis Conneau, Guillaume Lample, Conneau, Alexis +11 · 65 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1809.05053
openalex publication_date 2018/09/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in cross-lingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.
Citations
Cited by
- TCEval: Using Thermal Comfort to Assess Cognitive and Perceptual Abilities of AI
- Language Shapes Instruction Hierarchy Compliance in Multilingual LLMs
- Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
- Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structures
- Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM
- Oogiri-Master: Benchmarking Humor Understanding via Oogiri
- TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
- AraMix: Recycling, Refiltering, and Deduplicating to Deliver the Largest Arabic Pretraining Corpus
- Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models
- Easy Adaptation: An Efficient Task-Specific Knowledge Injection Method for Large Models in Resource-Constrained Environments
- BERnaT: Basque Encoders for Representing Natural Textual Diversity
- NormEval: A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization
- How Good is BLI as an Alignment Measure: A Study in Word Embedding Paradigm
- ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations
- Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages
- Routing-Based Continual Learning for Multimodal Large Language Models
- Simple Additions, Substantial Gains: Expanding Scripts, Languages, and Lineage Coverage in URIEL+
- Reward Models are Metrics in a Trench Coat
- Zero-Shot Cross-Lingual Transfer using Prefix-Based Adaptation
- AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
- PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models
- Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
- Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+
- Qomhra: A Bilingual Irish and English Large Language Model
- Vocab Diet: Reshaping the Vocabulary of LLMs with Vector Arithmetic
- ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
- Happiness is Sharing a Vocabulary: A Study of Transliteration Methods
- Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
- Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective
- Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
- Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B
- SocialNLI: A Dialogue-Centric Social Inference Dataset
- COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
- AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
- Aligning LLMs for Multilingual Consistency in Enterprise Applications
- Multilingual Vision-Language Models, A Survey
- False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
- Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
- On the Universality of Deep Contextual Language Models
- Quantifying Language Disparities in Multilingual Large Language Models
- Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges
- DeDisCo at the DISRPT 2025 Shared Task: A System for Discourse Relation Classification
- Continually Adding New Languages to Multilingual Language Models
- No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Models
- A Survey of Code-switched Speech and Language Processing
- Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkish
- L3Cube-IndicHeadline-ID: A Dataset for Headline Identification and Semantic Evaluation in Low-Resource Indian Languages
- FlauBERT: Unsupervised Language Model Pre-training for French
- Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
- Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
- Multilingual Datasets for Custom Input Extraction and Explanation Requests Parsing in Conversational XAI Systems
- Evaluating Multilingual and Code-Switched Alignment in LLMs via Synthetic Natural Language Inference
- KLUE: Korean Language Understanding Evaluation
- MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
- LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages
- SEA-BED: Southeast Asia Embedding Benchmark
- Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages
- Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
- A Comprehensive Survey of Multilingual Neural Machine Translation
- Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
- A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks
- EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
- Multilingual Political Views of Large Language Models: Identification and Steering
- Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation
Related