FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

2024/02/05 by Xing Han, Han, Xing, Huy V. Nguyen +7 · 9 citations
Computer Science · #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #Machine Learning (cs.LG) #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2402.03226

openalex publication_date 2024/02/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

Cited by

• Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification
• OmniField: Conditioned Neural Fields for Robust Multimodal Spatiotemporal Learning
• HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
• Guiding Mixture-of-Experts with Temporal Multimodal Interactions
• MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series
• TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraints
• A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing
• BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
• MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling

Related

• Meta Fusion: A Unified Framework For Multimodality Fusion with Mutual Learning
• MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
• I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
• MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
• FusionFM: All-in-One Multi-Modal Image Fusion with Flow Matching