2024/04/28 by Li Wan, Wan, Li, Tansu Alpcan +5
Computer Science · #Advanced Computational Techniques and Applications #Computation and Language (cs.CL) #Educational Technology and Assessment #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Text and Document Classification Technologies #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2405.01584
openalex publication_date 2024/04/28 · openalex created_date 2024/05/10 · openalex updated_date 2026/07/28
We propose a novel, lightweight supervised dictionary learning framework for text classification based on data compression and representation. This two-phase algorithm initially employs the Lempel-Ziv-Welch (LZW) algorithm to construct a dictionary from text datasets, focusing on the conceptual significance of dictionary elements. Subsequently, dictionaries are refined considering label data, optimizing dictionary atoms to enhance discriminative power based on mutual information and class distribution. This process generates discriminative numerical representations, facilitating the training of simple classifiers such as SVMs and neural networks. We evaluate our algorithm's information-theoretic performance using information bottleneck principles and introduce the information plane area rank (IPAR) as a novel metric to quantify the information-theoretic performance. Tested on six benchmark text datasets, our algorithm competes closely with top models, especially in limited-vocabulary contexts, using significantly fewer parameters. \reviewOur algorithm closely matches top-performing models, deviating by only ~2% on limited-vocabulary datasets, using just 10% of their parameters. However, it falls short on diverse-vocabulary datasets, likely due to the LZW algorithm's constraints with low-repetition data. This contrast highlights its efficiency and limitations across different dataset types.