An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification

2024/12/23 by Andre Rusli, Rusli, Andre, Makoto Shishido +1
Computer Science · #Sentiment Analysis and Opinion Mining #Topic Modeling #Advanced Text Analysis Techniques

paper · pdf · doi:10.48550/arxiv.2412.17361

Abstract

This study investigates the performance of three popular tokenization tools: MeCab, Sudachi, and SentencePiece, when applied as a preprocessing step for sentiment-based text classification of Japanese texts. Using Term Frequency-Inverse Document Frequency (TF-IDF) vectorization, we evaluate two traditional machine learning classifiers: Multinomial Naive Bayes and Logistic Regression. The results reveal that Sudachi produces tokens closely aligned with dictionary definitions, while MeCab and SentencePiece demonstrate faster processing speeds. The combination of SentencePiece, TF-IDF, and Logistic Regression outperforms the other alternatives in terms of classification performance.

Related

• An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese
• Japanese Sentiment Classification using a Tree-Structured Long Short-Term Memory with Attention
• USA: Universal Sentiment Analysis Model & Construction of Japanese Sentiment Text Classification and Part of Speech Dataset
• An Improved Text Sentiment Classification Model Using TF-IDF and Next Word Negation
• Text Classification in the LLM Era -- Where do we stand?