Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs

2025/08/04 by Yitong Zhu, Lei Han, Zhu, Yitong +6 · 2 citations
Computer Science · Psychology · #Emotion and Mood Recognition #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Mental Health via Writing #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2508.02133

openalex publication_date 2025/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal emotion recognition (MER) is crucial for human-computer interaction, yet real-world challenges like dynamic modality incompleteness and asynchrony severely limit its robustness. Existing methods often assume consistently complete data or lack dynamic adaptability. To address these limitations, we propose a novel Hi-MoE~(Hierarchical Mixture-of-Experts) framework for robust continuous emotion prediction. This framework employs a dual-layer expert structure. A Modality Expert Bank utilizes soft routing to dynamically handle missing modalities and achieve robust information fusion. A subsequent Emotion Expert Bank leverages differential-attention routing to flexibly attend to emotional prototypes, enabling fine-grained emotion representation. Additionally, a cross-modal alignment module explicitly addresses temporal shifts and semantic inconsistencies between modalities. Extensive experiments on benchmark datasets DEAP and DREAMER demonstrate our model's state-of-the-art performance in continuous emotion regression, showcasing exceptional robustness under challenging conditions such as dynamic modality absence and asynchronous sampling. This research significantly advances the development of intelligent emotion systems adaptable to complex real-world environments.

Citations

• Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
• StableMoE: Stable Routing Strategy for Mixture of Experts
• Mixture-of-Experts with Expert Choice Routing
• Scaling Vision with Sparse Mixture of Experts
• Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Recipes for building an open-domain chatbot
• M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Multi-attention Recurrent Network for Human Communication Comprehension
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Adaptive Mixtures of Local Experts

Cited by

• Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Experts
• BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition

Related

• More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment
• Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts
• GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints
• TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition
• ECMF: Enhanced Cross-Modal Fusion for Multimodal Emotion Recognition in MER-SEMI Challenge