TAGF: Time-aware Gated Fusion for Multimodal Valence-Arousal Estimation
2025/07/02 by Lee, Yubeen, Lee, Sangeun, Park, Chaewon +2
#FOS: Computer and information sciences #Multimedia (cs.MM) #Sound (cs.SD)
paper · doi:10.48550/arxiv.2507.02080
Abstract
Multimodal emotion recognition often suffers from performance degradation in valence-arousal estimation due to noise and misalignment between audio and visual modalities. To address this challenge, we introduce TAGF, a Time-aware Gated Fusion framework for multimodal emotion recognition. The TAGF adaptively modulates the contribution of recursive attention outputs based on temporal dynamics. Specifically, the TAGF incorporates a BiLSTM-based temporal gating mechanism to learn the relative importance of each recursive step and effectively integrates multistep cross-modal features. By embedding temporal awareness into the recursive fusion process, the TAGF effectively captures the sequential evolution of emotional expressions and the complex interplay between modalities. Experimental results on the Aff-Wild2 dataset demonstrate that TAGF achieves competitive performance compared with existing recursive attention-based models. Furthermore, TAGF exhibits strong robustness to cross-modal misalignment and reliably models dynamic emotional transitions in real-world conditions.
Citations
- DVD: A Comprehensive Dataset for Advancing Violence Detection in Real-World Scenarios
- MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network
- 7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition
- Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition
- Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition
- The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition
- Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond
- Multi-modal Facial Affective Analysis based on Masked Autoencoder
- ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges
- Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention
- ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges
- A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition
- ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges
- Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
- Analysing Affective Behavior in the second ABAW2 Competition
- Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
- Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
- Gated Mechanism for Attention Based Multimodal Sentiment Analysis
- Analysing Affective Behavior in the First ABAW 2020 Competition
- Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
- Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
- MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
- Deep Residual Learning for Image Recognition
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- United We Stand: Emphasizing Commonalities Across Cognitive-Behavioral Therapies
Related