LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models

2026/06/30 by Zhou Tao, Fang Zhang, Zewen Ding +5
Computer Science · #cs.CV

paper · pdf

arxiv created 2026/07/29 · arxiv updated 2026/07/30

Abstract

Multimodal Large Language Models (MLLMs) remain unreliable on fine-grained visual perception, even when high-resolution inputs preserve the necessary local details. We identify this limitation as visual context rot: decisive evidence may exist in the full image, yet fail to be reliably selected and used amid redundant visual context. We propose LOCUS (LOcal visual CUe Search), a training framework that teaches MLLMs to internalize local evidence search through a verifiable proxy task. During training, LOCUS provides a local crop as a visual cue and optimizes the model to recover its spatial support in the full image using an IoU-based reward. The visual cue is used only during training, leaving the standard image-question inference interface unchanged. Experiments across fine-grained perception, hallucination, general understanding, and reasoning benchmarks show that LOCUS improves localization-sensitive visual understanding while preserving broad capabilities. Attention analyses further indicate stronger focus on task-relevant evidence regions, suggesting that training-time visual cue search provides an effective route to internalized fine-grained evidence selection.

Citations

• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• Visual Jigsaw Post-Training Improves MLLMs
• HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
• Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning
• R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
• Visual-RFT: Visual Reinforcement Fine-Tuning
• MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?
• LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
• We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
• Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
• Modeling Context in Referring Expressions
• A Diagram Is Worth A Dozen Images
• Generation and Comprehension of Unambiguous Object Descriptions
• Microsoft COCO: Common Objects in Context

Related

• Mitigating Object Hallucination via Robust Local Perception Search
• Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
• VAR: Visual Attention Reasoning via Structured Search and Backtracking
• I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMs
• CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering