2024/05/21 by Hanlei Zhang, Zhang, Hanlei, Hua Xu +7 · 3 citations
Computer Science · Psychology · #Artificial intelligence #Cluster analysis #Computer science #Multimodal therapy #Natural Language Processing Techniques #Natural language processing #Programming language #Psychology #Semantic Web and Ontologies #Semantics (computer science) #Topic Modeling #cs.AI #cs.CL #cs.MM
paper · pdf · doi:10.48550/arxiv.2405.12775
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/05/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Discovering the semantics of multimodal utterances is essential for understanding human language and enhancing human-machine interactions. Existing methods manifest limitations in leveraging nonverbal information for discerning complex semantics in unsupervised scenarios. This paper introduces a novel unsupervised multimodal clustering method (UMC), making a pioneering contribution to this field. UMC introduces a unique approach to constructing augmentation views for multimodal data, which are then used to perform pre-training to establish well-initialized representations for subsequent clustering. An innovative strategy is proposed to dynamically select high-quality samples as guidance for representation learning, gauged by the density of each sample's nearest neighbors. Besides, it is equipped to automatically determine the optimal value for the top-K parameter in each cluster to refine sample selection. Finally, both high- and low-quality samples are used to learn representations conducive to effective clustering. We build baselines on benchmark multimodal intent and dialogue act datasets. UMC shows remarkable improvements of 2-7% scores in clustering metrics over state-of-the-art methods, marking the first successful endeavor in this domain. The complete code and data are available at https://github.com/thuiar/UMC.