Scope: Selective Cross-modal Orchestration of Visual Perception Experts

2025/10/14 by Tianyu Zhang, Zhang, Tianyu, Suyuchen Wang +13
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2510.12974

openalex publication_date 2025/10/14 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs.

Citations

• Ovis2.5 Technical Report
• DINOv3
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Qwen2.5-VL Technical Report
• LEO: Boosting Mixture of Vision Encoders for Multimodal Large Language Models
• BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks
• GPT-4o System Card
• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
• VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
• ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• MoVA: Adapting Mixture of Vision Experts to Multimodal Context
• Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
• BRAVE: Broadening the visual encoding of vision-language models
• LocCa: Visual Pretraining with Location-aware Captioners
• Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models
• mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding
• Mixtral of Experts
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
• DINOv2: Learning Robust Visual Features without Supervision
• Micrograph segmentations for DDEVD
• Segment Anything
• Sigmoid Loss for Language Image Pre-Training
• Prismer: A Vision-Language Model with Multi-Task Experts
• Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
• Mixture-of-Experts with Expert Choice Routing
• Masked Autoencoders Are Scalable Vision Learners
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision

Related

• MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
• Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts