A reproduction of Apple's bi-directional LSTM models for language identification in short strings

2021/02/11 by Toftrup, Mads, Sørensen, Søren Asger, Ciosici, Manuel R. +1 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2102.06282

Abstract

Language Identification is the task of identifying a document's language. For applications like automatic spell checker selection, language identification must use very short strings such as text message fragments. In this work, we reproduce a language identification architecture that Apple briefly sketched in a blog post. We confirm the bi-LSTM model's performance and find that it outperforms current open-source language identifiers. We further find that its language identification mistakes are due to confusion between related languages.

Cited by

• Improving Informally Romanized Language Identification

Related

• Automatic Language Identification in Texts: A Survey
• LIDE: Language Identification from Text Documents
• Native Language Identification with Large Language Models
• A Fast, Compact, Accurate Model for Language Identification of Codemixed Text
• A Simple and Efficient Probabilistic Language model for Code-Mixed Text