HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition

2021/08/07 by Ziyu Jia, Jia, Ziyu, Youfang Lin +9 · 15 citations
Computer Science · Neuroscience · Psychology · #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer science #Convolutional neural network #EEG and Brain-Computer Interfaces #Emotion and Mood Recognition #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #Graph #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Modal #Multimedia (cs.MM) #Pattern recognition (psychology) #Theoretical computer science #cs.AI #cs.HC #cs.LG #cs.MM

paper · pdf · doi:10.48550/arxiv.2108.03354

published in arXiv (Cornell University) (Cornell University) · Accepted by ACM MM 2021. The SOLE copyright holder is ACM Multimedia, all rights reserved

arxiv created 2021/08/07 · openalex publication_date 2021/08/07 · arxiv updated 2021/08/10 · openalex created_date 2021/09/13 · openalex updated_date 2026/08/08

Abstract

The research on human emotion under multimedia stimulation based on physiological signals is an emerging field, and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• Semi-Supervised Classification with Graph Convolutional Networks
• Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review
• SalientSleepNet: Multimodal Salient Wave Detection Network for Sleep Staging
• A Comprehensive Survey on Graph Neural Networks

Cited by

• LibEMER: A novel benchmark and algorithms library for EEG-based Multimodal Emotion Recognition
• TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graph
• Multi-View Spatial-Temporal Graph Convolutional Networks with Domain Generalization for Sleep Stage Classification
• EEGDM: Learning EEG Representation with Latent Diffusion Model
• Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions

Related

• Hierarchical Hypercomplex Network for Multimodal Emotion Recognition
• A Hybrid End-to-End Spatio-Temporal Attention Neural Network with Graph-Smooth Signals for EEG Emotion Recognition
• H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses
• PHemoNet: A Multimodal Network for Physiological Signals
• A Network-based Multimodal Data Fusion Approach for Characterizing Dynamic Multimodal Physiological Patterns