V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
2023/12/21 by Penghao Wu, Saining Xie, Wu, Penghao +1 · 2 voices · 132 citations
Computer Science · #Multimodal Machine Learning Applications #Semantic Web and Ontologies #Image Retrieval and Classification Techniques
paper · pdf · doi:10.48550/arxiv.2312.14135
Abstract
When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.
Cited by
- RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models
- Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints
- Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
- ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding
- Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
- AdaTurn: Budget-Aware Test-Time Scaling for Active Visual Perception Agents
- GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
- Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
- LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
- RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
- LanteRn: Latent Visual Structured Reasoning
- Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding
- A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
- CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
- Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images
- Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
- Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
- Step-GUI Technical Report
- ViRC: Enhancing Visual Interleaved Mathematical CoT with Reason Chunking
- OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
- A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments
- AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
- DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
- Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
- Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
- Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models
- ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
- InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- Thinking with Images via Self-Calling Agent
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
- START: Spatial and Textual Learning for Chart Understanding
- Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
- Training Multi-Image Vision Agents via End2End Reinforcement Learning
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
- MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
- Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch
- VACoT: Rethinking Visual Data Augmentation with VLMs
- Artemis: Structured Visual Reasoning for Perception Policy Learning
- From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
- Unexplored flaws in multiple-choice VQA evaluations
- Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning
- Qwen3-VL Technical Report
- Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search
- Thinking in 360°: Humanoid Visual Search in the Wild
- Boosting Reasoning in Large Multimodal Models via Activation Replay
- CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception
- CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
- LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
- Understanding Task Transfer in Vision-Language Models
- Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
- ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better
- VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
- Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
- Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration
- Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning
- Taming Object Hallucinations with Verified Atomic Confidence Estimation
- SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale
- DeepEyesV2: Toward Agentic Multimodal Model
- TRACE: Textual Reasoning for Affordance Coordinate Extraction
- LaRe: Latent Refocusing for Multimodal Reasoning
- SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
- TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
- Generating Accurate and Detailed Captions for High-Resolution Images
- Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
- Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
- SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
- On the Faithfulness of Visual Thinking: Measurement and Enhancement
- CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
- Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
- FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
- Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation
- I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs
- VAR: Visual Attention Reasoning via Structured Search and Backtracking
- Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
- SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
- Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
- A Survey on Agentic Multimodal Large Language Models
- FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model
- Task-Aware Resolution Optimization for Visual Large Language Models
- Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
- What MLLMs Learn about When they Learn about Multimodal Reasoning
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
- Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models
- Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
- TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understanding
- VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
- From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
- Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models
- ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
- Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models
- VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
- FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification
- Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
- Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
- Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
- BaseReward: A Strong Baseline for Multimodal Reward Model
- UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets
- AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
- EZREAL: Enhancing Zero-Shot Outdoor Robot Navigation toward Distant Targets under Varying Visibility
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
- Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
- VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
- LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
- MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
- Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
- Video-LLMs with Temporal Visual Screening
- Simple o3: Towards Interleaved Vision-Language Reasoning
- Thyme: Think Beyond Images
- OpenCUA: Open Foundations for Computer-Use Agents
- VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding
- Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning
- Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
- Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
- Evaluating Variance in Visual Question Answering Benchmarks
- A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
Discussions
Related