BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models

2025/10/22 by Gao, Yuan, Salhan, Suchir, Caines, Andrew +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.19419

Abstract

To bridge the gap between performance-oriented benchmarks and the evaluation of cognitively inspired models, we introduce BLiSS 1.0, a Benchmark of Learner Interlingual Syntactic Structure. Our benchmark operationalizes a new paradigm of selective tolerance, testing whether a model finds a naturalistic learner error more plausible than a matched, artificial error within the same sentence. Constructed from over 2.8 million naturalistic learner sentences, BLiSS provides 136,867 controlled triplets (corrected, learner, artificial) for this purpose. Experiments on a diverse suite of models demonstrate that selective tolerance is a distinct capability from standard grammaticality, with performance clustering strongly by training paradigm. This validates BLiSS as a robust tool for measuring how different training objectives impact a model's alignment with the systematic patterns of human language acquisition.

Citations

• Minimal Pair-Based Evaluation of Code-Switching
• On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
• MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series
• Release of Pre-Trained Models for the Japanese Language
• CroissantLLM: A Truly Bilingual French-English Language Model
• Investigating Critical Period Effects in Language Acquisition through Neural Language Models
• Language acquisition: do children and language models follow similar learning stages?
• Second Language Acquisition of Neural Language Models
• SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction

Related

• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
• Targeted Syntactic Evaluation of Language Models
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition
• Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models
• Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding