MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

2025/04/03 by Jaap Jumelet, Leonie Weissweiler, Jumelet, Jaap +5 · 2 voices · 28 citations
Computer Science · Social Sciences · #Benchmark (surveying) #Computational linguistics #Language and cultural evolution #Massively parallel #Natural Language Processing Techniques #Natural language #Topic Modeling

paper · pdf · open access · doi:10.48550/arxiv.2504.02768

published in University of Groningen research database (University of Groningen / Centre for Information Technology) (University of Groningen)

openalex publication_date 2025/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

Citations

• Language Models Fail to Introspect About Their Knowledge of Language
• Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs
• 2 OLMo 2 Furious
• Natural Language Processing RELIES on Linguistics
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Linear representations of grammaticality in neural language models
• Different types of syntactic agreement recruit the same units within large language models
• BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data
• Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
• Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs
• A Study of Crosslinguistic Influence in Language Models
• Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation
• DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
• Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis
• On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
• Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
• Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting
• COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
• QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
• Multilingual Vision-Language Models, A Survey
• Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
• Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect
• Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining
• Understanding Syntactic Generalization in Structure-inducing Language Models
• UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu
• Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken English
• Evaluating Morphological Alignment of Tokenizers in 70 Languages
• Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations
• TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs
• Minimal Pair-Based Evaluation of Code-Switching
• UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions
• MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages

Discussions

• I'm in Suzhou to present our work on MultiBLiMP, Friday @ 11:45 in the Multilinguality session (A301)! Come check it out if your interested in multilingual linguistic evaluation of LLMs (there will be [bsky, 27 points, 0 comments]
• We see our work as a (breadth-first) complement to the valuable development of language-specific BLiMPs which offer more depth but are hardly scalable to lots of languages. For more details and intere [bsky, 3 points, 0 comments]

Related

• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs
• QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
• RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
• XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs