2021/11/10 by Lev Evtodienko, Evtodienko, Lev
Computer Science · Psychology · #Audio and Speech Processing (eess.AS) #Computer Vision and Pattern Recognition (cs.CV) #Emotion and Mood Recognition #FOS: Computer and information sciences #FOS: Electrical engineering #Human Pose and Action Recognition #Music and Audio Processing #Sound (cs.SD) #Speech and Audio Processing #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2111.05890
openalex publication_date 2021/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Classifying group-level emotions is a challenging task due to complexity of video, in which not only visual, but also audio information should be taken into consideration. Existing works on multimodal emotion recognition are using bulky approach, where pretrained neural networks are used as a feature extractors and then extracted features are being fused. However, this approach does not consider attributes of multimodal data and feature extractors cannot be fine-tuned for specific task which can be disadvantageous for overall model accuracy. To this end, our impact is twofold: (i) we train model end-to-end, which allows early layers of neural network to be adapted with taking into account later, fusion layers, of two modalities; (ii) all layers of our model was fine-tuned for downstream task of emotion recognition, so there were no need to train neural networks from scratch. Our model achieves best validation accuracy of 60.37% which is approximately 8.5% higher, than VGAF dataset baseline and is competitive with existing works, audio and video modalities.