Language Identification in Code-Mixed Data using Multichannel Neural\n Networks and Context Capture

2018/08/21 by Soumil Mandal, Mandal, Soumil, Anil Kumar Singh +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Natural Language Processing Techniques #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.1808.07118

openalex publication_date 2018/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

An accurate language identification tool is an absolute necessity for\nbuilding complex NLP systems to be used on code-mixed data. Lot of work has\nbeen recently done on the same, but there's still room for improvement.\nInspired from the recent advancements in neural network architectures for\ncomputer vision tasks, we have implemented multichannel neural networks\ncombining CNN and LSTM for word level language identification of code-mixed\ndata. Combining this with a Bi-LSTM-CRF context capture module, accuracies of\n93.28% and 93.32% is achieved on our two testing sets.\n

Related

• Byte-based Language Identification with Deep Convolutional Networks
• Insights into End-to-End Learning Scheme for Language Identification
• Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
• A Fast, Compact, Accurate Model for Language Identification of Codemixed Text
• DIVERS-Bench: Evaluating Language Identification Across Domain Shifts and Code-Switching