2022/12/20 by Feng Qiu, Qiu, Feng, Wanzeng Kong +3
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Emotion and Mood Recognition #FOS: Computer and information sciences #Sentiment Analysis and Opinion Mining
paper · pdf · doi:10.48550/arxiv.2212.10030
openalex publication_date 2022/12/20 · openalex created_date 2023/01/04 · openalex updated_date 2026/07/28
Humans are sophisticated at reading interlocutors' emotions from multimodal signals, such as speech contents, voice tones and facial expressions. However, machines might struggle to understand various emotions due to the difficulty of effectively decoding emotions from the complex interactions between multimodal signals. In this paper, we propose a multimodal emotion analysis framework, InterMulti, to capture complex multimodal interactions from different views and identify emotions from multimodal signals. Our proposed framework decomposes signals of different modalities into three kinds of multimodal interaction representations, including a modality-full interaction representation, a modality-shared interaction representation, and three modality-specific interaction representations. Additionally, to balance the contribution of different modalities and learn a more informative latent interaction representation, we developed a novel Text-dominated Hierarchical High-order Fusion(THHF) module. THHF module reasonably integrates the above three kinds of representations into a comprehensive multimodal interaction representation. Extensive experimental results on widely used datasets, (i.e.) MOSEI, MOSI and IEMOCAP, demonstrate that our method outperforms the state-of-the-art.