Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation
2025/09/17 by Xiaobo Yang, Yang, Xiaobo, Xiaojin Gong +1
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2509.13676
openalex publication_date 2025/09/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Recently, Referring Image Segmentation (RIS) frameworks that pair the Multimodal Large Language Model (MLLM) with the Segment Anything Model (SAM) have achieved impressive results. However, adapting MLLM to segmentation is computationally intensive, primarily due to visual token redundancy. We observe that traditional patch-wise visual projectors struggle to strike a balance between reducing the number of visual tokens and preserving semantic clarity, often retaining overly long token sequences to avoid performance drops. Inspired by text tokenizers, we propose a novel semantic visual projector that leverages semantic superpixels generated by SAM to identify "visual words" in an image. By compressing and projecting semantic superpixels as visual tokens, our approach adaptively shortens the token sequence according to scene complexity while minimizing semantic loss in compression. To mitigate loss of information, we propose a semantic superpixel positional embedding to strengthen MLLM's awareness of superpixel geometry and position, alongside a semantic superpixel aggregator to preserve both fine-grained details inside superpixels and global context outside. Experiments show that our method cuts visual tokens by 93% without compromising performance, notably speeding up MLLM training and inference, and outperforming existing compressive visual projectors on RIS.
Citations
- QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
- InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression
- MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
- Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
- Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition
- Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- TokenPacker: Efficient Visual Projector for Multimodal LLM
- OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Honeybee: Locality-enhanced Projector for Multimodal LLM
- PixelLM: Pixel Reasoning with Large Multimodal Model
- PerceptionGPT: Effectively Fusing Visual Perception into LLM
- Ferret: Refer and Ground Anything Anywhere at Any Granularity
- Improved Baselines with Visual Instruction Tuning
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
- Towards Complex-query Referring Image Segmentation: A Novel Benchmark
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- Kosmos-2: Grounding Multimodal Large Language Models to the World
- Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
- Waffling around for Performance: Visual Classification with Random Words and Broad Concepts
- VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
- Visual Instruction Tuning
- DINOv2: Learning Robust Visual Features without Supervision
- Segment Everything Everywhere All at Once
- Micrograph segmentations for DDEVD
- Segment Anything
- Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
- PolyFormer: Referring Image Segmentation as Sequential Polygon Generation
- Generalized Decoding for Pixel, Image, and Language
- VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Flamingo: a Visual Language Model for Few-Shot Learning
- How well does CLIP understand texture?
- Masked-attention Mask Transformer for Universal Image Segmentation
- LoRA: Low-Rank Adaptation of Large Language Models
- Referring Transformer: A One-step Approach to Multi-task Visual Grounding
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Learning Transferable Visual Models From Natural Language Supervision
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- End-to-End Object Detection with Transformers
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Attention Is All You Need
- Modeling Context in Referring Expressions
- Generation and Comprehension of Unambiguous Object Descriptions
- Neural Machine Translation of Rare Words with Subword Units
- SLIC Superpixels Compared to State-of-the-Art Superpixel Methods
- RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Related