Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

2023/12/01 by Jialin Wu, Hu Xia, Wu, Jialin +7 · 9 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2312.00968

openalex publication_date 2023/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. However, generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. Recent research suggests that Mixture of Experts (MoE) architectures are useful for instruction tuning, but for LMMs of parameter size around O(50-100B), the prohibitive cost of replicating and storing the expert models severely limits the number of experts we can use. We propose Omni-SMoLA, an architecture that uses the Soft MoE approach to (softly) mix many multimodal low rank experts, and avoids introducing a significant number of new parameters compared to conventional MoE models. The core intuition here is that the large model provides a foundational backbone, while different lightweight experts residually learn specialized knowledge, either per-modality or multimodally. Extensive experiments demonstrate that the SMoLA approach helps improve the generalist performance across a broad range of generative vision-and-language tasks, achieving new SoTA generalist performance that often matches or outperforms single specialized LMM baselines, as well as new SoTA specialist performance.

Cited by

• How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
• Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification
• Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
• MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
• ViFP: A Framework for Visual False Positive Detection to Enhance Reasoning Reliability in VLMs
• Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs
• Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
• Mixture of Experts in Large Language Models

Related

• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
• MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
• Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities