Are All Languages Equally Hard to Language-Model?

2018/06/10 by Cotterell, Ryan, Mielke, Sabrina J., Eisner, Jason +1 · 7 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1806.03743

Abstract

For general modeling methods applied to diverse languages, a natural question is: how well should we expect our models to work on languages with differing typological profiles? In this work, we develop an evaluation framework for fair cross-linguistic comparison of language models, using translated text so that all models are asked to predict approximately the same information. We then conduct a study on 21 languages, demonstrating that in some languages, the textual expression of the information is harder to predict with both n-gram and LSTM language models. We show complex inflectional morphology to be a cause of performance differences among languages.

Cited by

• Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
• Form and Meaning in Intrinsic Multilingual Evaluations
• Confounding Factors in Relating Model Performance to Morphology
• Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention
• Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages
• MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment

Related

• What Kind of Language Is Hard to Language-Model?
• Why do language models perform worse for morphologically complex languages?
• Multilingual Large Language Models do not comprehend all natural languages to equal degrees
• Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models
• Quantifying Language Disparities in Multilingual Large Language Models