Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

2024/05/18 by Yunxin Li, Li, Yunxin, Shenyuan Jiang +13 · 50 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimedia (cs.MM) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.11273

openalex publication_date 2024/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs.

Cited by

• Universal Pansharpening Model
• Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation
• Metacognitive Sensitivity for Test-Time Dynamic Model Selection
• AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
• Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
• Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification
• Route Experts by Sequence, not by Token
• BrainCSD: A Hierarchical Consistency-Driven MoE Foundation Model for Unified Connectome Synthesis and Multitask Brain Trait Prediction
• MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts
• ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
• End-to-end Listen, Look, Speak and Act
• UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE
• A Survey on Agentic Multimodal Large Language Models
• Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation
• Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems
• MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series
• PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning
• SAIL-VL2 Technical Report
• Too Helpful, Too Harmless, Too Honest or Just Right?
• MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security
• Decoding Visual Neural Representations by Multimodal with Dynamic Balancing
• Insight-LLM: LLM-enhanced Multi-view Fusion in Insider Threat Detection
• BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
• Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models
• S2M3: Split-and-Share Multi-Modal Models for Distributed Multi-Task Inference on the Edge
• TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding
• M3LLM: Model Context Protocol-aided Mixture of Vision Experts For Multimodal LLMs in Networks
• Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edge
• Neural Inhibition Improves Dynamic Routing and Mixture of Experts
• DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
• MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models
• AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generation
• GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models
• ComfyUI-R1: Exploring Reasoning Models for Workflow Generation
• SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
• SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarios
• InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
• Farm-LightSeek: An Edge-centric Multimodal Agricultural IoT Data Analytics Framework with Lightweight LLMs
• Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilities
• Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
• VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization
• FractalMamba++: Scaling Vision Mamba Across Resolutions via Hilbert Fractal Geometry
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
• Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving
• TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training
• Mixture of Experts for Decentralized Generative AI and Reinforcement Learning in Wireless Networks: A Comprehensive Survey
• Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
• MME: Mixture of Mesh Experts with Random Walk Transformer Gating
• VideoVista-CulturalLingo: 360^∘ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension
• NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation

Related

• CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
• Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
• MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
• MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
• MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning