Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

2018/05/22 by Gu, Yue, Yang, Kangning, Fu, Shiyu +3 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.1805.08660

Abstract

Multimodal affective computing, learning to recognize and interpret human affects and subjective information from multiple data sources, is still challenging because: (i) it is hard to extract informative features to represent human affects from heterogeneous inputs; (ii) current fusion strategies only fuse different modalities at abstract level, ignoring time-dependent interactions between modalities. Addressing such issues, we introduce a hierarchical multimodal architecture with attention and word-level fusion to classify utter-ance-level sentiment and emotion from text and audio data. Our introduced model outperforms the state-of-the-art approaches on published datasets and we demonstrated that our model is able to visualize and interpret the synchronized attention over modalities.

Cited by

• CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection

Related

• Rethinking Multimodal Sentiment Analysis: A High-Accuracy, Simplified Fusion Architecture
• Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features
• Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification
• Multi-Modal Sentiment Analysis with Dynamic Attention Fusion
• Heterogeneous bimodal attention fusion for speech emotion recognition