HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition

2021/08/07 by Ziyu Jia, Jia, Ziyu, Youfang Lin +9 · 4 citations
Computer Science · Neuroscience · Psychology · #Artificial Intelligence (cs.AI) #EEG and Brain-Computer Interfaces #Emotion and Mood Recognition #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Multimedia (cs.MM)

paper · pdf · doi:10.48550/arxiv.2108.03354

openalex publication_date 2021/08/07 · openalex created_date 2021/09/13 · openalex updated_date 2026/07/28

Abstract

The research on human emotion under multimedia stimulation based on physiological signals is an emerging field, and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• Semi-Supervised Classification with Graph Convolutional Networks
• Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
• SalientSleepNet: Multimodal Salient Wave Detection Network for Sleep Staging
• A Comprehensive Survey on Graph Neural Networks

Cited by

• LibEMER: A novel benchmark and algorithms library for EEG-based Multimodal Emotion Recognition
• TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph
• Multi-View Spatial-Temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification
• EEGDM: Learning EEG Representation with Latent Diffusion Model

Related

• Hierarchical Hypercomplex Network for Multimodal Emotion Recognition
• A Hybrid End-to-End Spatio-Temporal Attention Neural Network with Graph-Smooth Signals for EEG Emotion Recognition
• H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses
• PHemoNet: A Multimodal Network for Physiological Signals
• A Network-based Multimodal Data Fusion Approach for Characterizing Dynamic Multimodal Physiological Patterns