2020/02/01 by Sourabh Vasant Gothe, Sourav Ghosh, Sharmila Mani +3
Computer Science · #Ambiguity #Artificial intelligence #Character (mathematics) #Code (set theory) #Computer science #Inference #Language identification #Language model #Mobile device #Natural Language Processing Techniques #Natural language #Natural language processing #Programming language #Speech Recognition and Synthesis #Speech and dialogue systems #Speech recognition #cs.CL
paper · pdf · doi:10.1109/icsc.2020.00057
published as 2020 IEEE 14th International Conference on Semantic Computing (ICSC), San Diego, CA, USA, 2020, pp. 279-286 · 2020 IEEE 14th International Conference on Semantic Computing (ICSC). Accessible at https://ieeexplore.ieee.org/document/9031474
openalex publication_date 2020/02/01 · arxiv created 2021/01/07 · arxiv updated 2021/01/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
More than 2 billion mobile users worldwide type in multiple languages in the soft keyboard. On a monolingual keyboard, 38% of falsely auto-corrected words are valid in another language. This can be easily avoided by detecting the language of typed words and then validating it in its respective language. Language detection is a well-known problem in natural language processing. In this paper, we present a fast, light-weight and accurate Language Detection Engine (LDE) for multilingual typing that dynamically adapts to user intended language in realtime. We propose a novel approach where the fusion of character N-gram model [1] and logistic regression [2] based selector model is used to identify the language. Additionally, we present a unique method of reducing the inference time significantly by parameter reduction technique. We also discuss various optimizations fabricated across LDE to resolve ambiguity in input text among the languages with the same character pattern. Our method demonstrates an average accuracy of 94.5% for Indian languages in Latin script and that of 98% for European languages on the code-switched data. This model outperforms fastText [3] by 60.39% and ML-Kit1 by 23.67% in F1 score [4] for European languages. LDE is faster on mobile device with an average inference time of 25.91 micro seconds.