2024/10/04 by Pritom Saha Akash, Akash, Pritom Saha, Kevin Chen–Chuan Chang +1 · 2 citations
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #Computer science #Context (archaeology) #FOS: Computer and information sciences #History #Information Retrieval (cs.IR) #Linguistics #Natural Language Processing Techniques #Natural language processing #Philosophy #Prefix #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2410.03071
openalex publication_date 2024/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Topic modeling is a powerful technique for uncovering hidden themes within a collection of documents. However, the effectiveness of traditional topic models often relies on sufficient word co-occurrence, which is lacking in short texts. Therefore, existing approaches, whether probabilistic or neural, frequently struggle to extract meaningful patterns from such data, resulting in incoherent topics. To address this challenge, we propose a novel approach that leverages large language models (LLMs) to extend short texts into more detailed sequences before applying topic modeling. To further improve the efficiency and solve the problem of semantic inconsistency from LLM-generated texts, we propose to use prefix tuning to train a smaller language model coupled with a variational autoencoder for short-text topic modeling. Our method significantly improves short-text topic modeling performance, as demonstrated by extensive experiments on real-world datasets with extreme data sparsity, outperforming current state-of-the-art topic models.