Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation

2025/11/14 by Trinh, Quoc-Huy
Computer Science · #Computational complexity theory #Computational model #Generative Adversarial Networks and Image Synthesis #Key (lock) #Limit (mathematics) #Modulation (music) #Multimodal Machine Learning Applications #Quadratic equation #Software deployment #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.11177

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/11/14 · openalex created_date 2025/11/18 · openalex updated_date 2026/08/05

Abstract

Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in vision-language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as robotic manipulation, personal assistants, and smart cameras. Most existing methods rely on Transformer-based cross-attention, whose quadratic complexity hinders efficiency. Moreover, small vision-language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Viper-F1, a Hybrid State-Space Vision-Language Model that replaces attention with efficient Liquid State-Space Dynamics. To further enhance visual grounding, we propose a Token-Grid Correlation Module, which computes lightweight correlations between text tokens and image patches and modulates the state-space dynamics via FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual prompt while maintaining linear-time inference. Experimental results across multiple benchmarks demonstrate that Viper-F1 achieves accurate, fine-grained understanding with significantly improved efficiency.

Citations

• SmolVLM: Redefining small and efficient multimodal models
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Liquid: Language Models are Scalable and Unified Multi-modal Generators
• Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement
• LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
• Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
• MobileVLM: A Vision-Language Model for Better Intra- and Inter-UI Understanding
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
• Chameleon: Mixed-Modal Early-Fusion Foundation Models
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
• MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
• LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
• Improved Baselines with Visual Instruction Tuning
• Qwen Technical Report
• OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
• MMBench: Is Your Multi-modal Model an All-around Player?
• Kosmos-2: Grounding Multimodal Large Language Models to the World
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
• Evaluating Object Hallucination in Large Vision-Language Models
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• Liquid Structural State-Space Models
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• On the Parameterization and Initialization of Diagonal State Space Models
• Efficiently Modeling Long Sequences with Structured State Spaces
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Understanding Guided Image Captioning Performance across Domains
• FiLM: Visual Reasoning with a General Conditioning Layer
• Attention Is All You Need
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• A Diagram Is Worth A Dozen Images

Related

• Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
• Liquid: Language Models are Scalable and Unified Multi-modal Generators
• Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
• LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation