2013/05/03 by Deqing Wang, Hui Zhang, Wang, Deqing +5
Computer Science · Mathematics · #Advanced Text Analysis Techniques #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Text and Document Classification Technologies #Web Data Mining and Analysis #cs.IR #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1305.0638
5pages 9 figures CIKM2012 paper
arxiv created 2013/05/03 · openalex publication_date 2013/05/03 · arxiv updated 2013/05/06 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28
Much work has been done on feature selection. Existing methods are based on document frequency, such as Chi-Square Statistic, Information Gain etc. However, these methods have two shortcomings: one is that they are not reliable for low-frequency terms, and the other is that they only count whether one term occurs in a document and ignore the term frequency. Actually, high-frequency terms within a specific category are often regards as discriminators. This paper focuses on how to construct the feature selection function based on term frequency, and proposes a new approach based on t-test, which is used to measure the diversity of the distributions of a term between the specific category and the entire corpus. Extensive comparative experiments on two text corpora using three classifiers show that our new approach is comparable to or or slightly better than the state-of-the-art feature selection methods (i.e., χ2, and IG) in terms of macro-F1 and micro-F1.