2023/12/06 by Arinjay Wyawhare, Wyawhare, Arinjay · 2 citations
Computer Science · #Computation and Language (cs.CL) #Data Mining and Machine Learning Applications #Edcuational Technology Systems #FOS: Computer and information sciences #Text and Document Classification Technologies
paper · pdf · doi:10.48550/arxiv.2312.03789
openalex publication_date 2023/12/06 · openalex created_date 2023/12/09 · openalex updated_date 2026/07/28
This research conducts a comparative study on multilingual text classification methods, utilizing deep learning and embedding visualization. The study employs LangDetect, LangId, FastText, and Sentence Transformer on a dataset encompassing 17 languages. It explores dimensionality's impact on clustering, revealing FastText's clearer clustering in 2D visualization due to its extensive multilingual corpus training. Notably, the FastText multi-layer perceptron model achieved remarkable accuracy, precision, recall, and F1 score, outperforming the Sentence Transformer model. The study underscores the effectiveness of these techniques in multilingual text classification, emphasizing the importance of large multilingual corpora for training embeddings. It lays the groundwork for future research and assists practitioners in developing language detection and classification systems. Additionally, it includes the comparison of multi-layer perceptron, LSTM, and Convolution models for classification.