V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
2023/12/21 by Penghao Wu, Saining Xie, Wu, Penghao +1 · 2 voices · 215 citations
Computer Science · #Artificial intelligence #Benchmark (surveying) #Computer science #Focus (optics) #Geography #Human–computer interaction #Image Retrieval and Classification Techniques #Mechanism (biology) #Multimodal Machine Learning Applications #Process (computing) #Programming language #Semantic Web and Ontologies #Visual search
paper · pdf · doi:10.48550/arxiv.2312.14135
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/12/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
When we look around and perform complex tasks, how we see and selectively process what we see is crucial. However, the lack of this visual search mechanism in current multimodal LLMs (MLLMs) hinders their ability to focus on important visual details, especially when handling high-resolution and visually crowded images. To address this, we introduce V*, an LLM-guided visual search mechanism that employs the world knowledge in LLMs for efficient visual querying. When combined with an MLLM, this mechanism enhances collaborative reasoning, contextual understanding, and precise targeting of specific visual elements. This integration results in a new MLLM meta-architecture, named Show, sEArch, and TelL (SEAL). We further create V*Bench, a benchmark specifically designed to evaluate MLLMs in their ability to process high-resolution images and focus on visual details. Our study highlights the necessity of incorporating visual search capabilities into multimodal systems. The code is available https://github.com/penghao-wu/vstar.
Cited by
- RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models
- Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints
- Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
- ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding
- Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs
- AdaTurn: Budget-Aware Test-Time Scaling for Active Visual Perception Agents
- GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
- LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
- Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
- LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
- RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
- LanteRn: Latent Visual Structured Reasoning
- Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding
- A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
- CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
- Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images
- Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
- Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
- Step-GUI Technical Report
- ViRC: Enhancing Visual Interleaved Mathematical CoT with Reason Chunking
- OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving
- A Deep Learning Model of Mental Rotation Informed by Interactive VR Experiments
- AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
- DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
- Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
- Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
- Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models
- ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
- InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- Thinking with Images via Self-Calling Agent
- No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
- START: Spatial and Textual Learning for Chart Understanding
- Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
- Training Multi-Image Vision Agents via End2End Reinforcement Learning
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
- MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
- GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
- Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch
- VACoT: Rethinking Visual Data Augmentation with VLMs
- Artemis: Structured Visual Reasoning for Perception Policy Learning
- From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning
- Unexplored flaws in multiple-choice VQA evaluations
- Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning
- Qwen3-VL Technical Report
- Look Where It Matters: Training-Free Ultra-HR Remote Sensing VQA via Adaptive Zoom Search
- Thinking in 360°: Humanoid Visual Search in the Wild
- Boosting Reasoning in Large Multimodal Models via Activation Replay
- CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception
- CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
- LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
- Understanding Task Transfer in Vision-Language Models
- Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
- ChainV: Atomic Visual Hints Make Multimodal Reasoning Shorter and Better
- VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
- Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
- Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration
- Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning
- Taming Object Hallucinations with Verified Atomic Confidence Estimation
- SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale
- DeepEyesV2: Toward Agentic Multimodal Model
- TRACE: Textual Reasoning for Affordance Coordinate Extraction
- LaRe: Latent Refocusing for Multimodal Reasoning
- SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
- VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
- TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
- Generating Accurate and Detailed Captions for High-Resolution Images
- Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
- Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
- SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
- SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
- On the Faithfulness of Visual Thinking: Measurement and Enhancement
- CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
- Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
- FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
- Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculation
- I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs
- VAR: Visual Attention Reasoning via Structured Search and Backtracking
- Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
- SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
- Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
- A Survey on Agentic Multimodal Large Language Models
- FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model
- Task-Aware Resolution Optimization for Visual Large Language Models
- Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning
- What MLLMs Learn about When they Learn about Multimodal Reasoning
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
- Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models
- Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs
- TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understanding
- VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
- From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
- Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
- HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
- Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
- Towards Faithful Reasoning in Remote Sensing: A Perceptually-Grounded GeoSpatial Chain-of-Thought for Vision-Language Models
- ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
- Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
- CoFFT: Chain of Foresight-Focus Thought for Visual Language Models
- VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
- FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification
- Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA
- Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perception
- Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
- BaseReward: A Strong Baseline for Multimodal Reward Model
- UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets
- AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity
- EZREAL: Enhancing Zero-Shot Outdoor Robot Navigation toward Distant Targets under Varying Visibility
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search
- Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
- VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use
- LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
- MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents
- Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
- Video-LLMs with Temporal Visual Screening
- Simple o3: Towards Interleaved Vision-Language Reasoning
- Thyme: Think Beyond Images
- OpenCUA: Open Foundations for Computer-Use Agents
- VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding
- Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
- SIFThinker: Spatially-Aware Image Focus for Visual Reasoning
- Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
- Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
- Evaluating Variance in Visual Question Answering Benchmarks
- A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models
- Eyes Will Shut: A Vision-Based Next GPS Location Prediction Model by Reinforcement Learning from Visual Map Feed Back
- Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
- Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
- Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs
- PoseLLM: Enhancing Language-Guided Human Pose Estimation with MLP Alignment
- PyVision: Agentic Vision with Dynamic Tooling
- Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology
- High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning
- LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance
- R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding
- AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
- How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks
- Kwai Keye-VL Technical Report
- Look-Back: Implicit Visual Re-focusing in MLLM Reasoning
- Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints
- UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding
- Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation
- LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
- Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
- DAVID-XR1: Detecting AI-Generated Videos with Explainable Reasoning
- VGR: Visual Grounded Reasoning
- DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations
- LUT: Latent Utility Training for Visual Reasoning
- MiMo-VL Technical Report
- Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement
- GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
- Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning
- Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research
- AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time
- SORCE: Small Object Retrieval in Complex Environments
- Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought
- Grounded Reinforcement Learning for Visual Reasoning
- VModA: An Effective Framework for Adaptive NSFW Image Moderation
- QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
- ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning
- Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
- CROP: Contextual Region-Oriented Visual Token Pruning
- Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
- Increasing Computation Resolves Conflicts in Vision Language Models
- v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning
- ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps
- One RL to See Them All: Visual Triple Unified Reinforcement Learning
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
- VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
- Let Androids Dream of Electric Sheep: A Human-Inspired Image Implication Understanding and Reasoning Framework
- Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
- Dynamic Resolution Routing for Efficient Egocentric Grounding
- Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
- ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
- Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
- Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
- OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
- DSADF: Thinking Fast and Slow for Decision Making
- What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs
- SITE: towards Spatial Intelligence Thorough Evaluation
- Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images
- Grounding Task Assistance with Multimodal Cues from a Single Demonstration
- FINER: MLLMs Hallucinate under Fine-grained Negative Queries
- ETCHR: Editing To Clarify and Harness Reasoning
- Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
- Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
- Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
- LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
- SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs
- RegionReasoner: Region-Grounded Multi-Round Visual Reasoning
- HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?
- Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization
- Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models
- Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning
- Evaluating Graphical Perception with Multimodal LLMs
- Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
- Describe Anything: Detailed Localized Image and Video Captioning
- DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understanding
- LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception
- AGI Is Coming... Right After AI Learns to Play Wordle
- Towards Explainable Fake Image Detection with Multi-Modal Large Language Models
- TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection
- Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering
Discussions
Related