2020/04/30 by Maharshi R. Pandya, Pandya, Maharshi R., Jessica Wolpaw Reyes +3
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2005.00042
openalex publication_date 2020/04/30 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Using author provided tags to predict tags for a new document often results\nin the overgeneration of tags. In the case where the author doesn't provide any\ntags, our documents face the severe under-tagging issue. In this paper, we\npresent a method to generate a universal set of tags that can be applied widely\nto a large document corpus. Using IBM Watson's NLU service, first, we collect\nkeywords/phrases that we call "complex document tags" from 8,854 popular\nreports in the corpus. We apply LDA model over these complex document tags to\ngenerate a set of 765 unique "simple tags". In applying the tags to a corpus of\ndocuments, we run each document through the IBM Watson NLU and apply\nappropriate simple tags. Using only 765 simple tags, our method allows us to\ntag 87,397 out of 88,583 total documents in the corpus with at least one tag.\nAbout 92.1% of the total 87,397 documents are also determined to be\nsufficiently-tagged. In the end, we discuss the performance of our method and\nits limitations.\n