One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting

2026/07/30 by Rui Tang, Wentao Yang, Peirong Zhang +4
Computer Science · #cs.CV

paper · pdf · doi:10.1145/3767308.3835174

published as Proceedings of the 34th ACM International Conference on Multimedia (MM '26), 2026 · 15 pages, 11 figures. Accepted to ACM Multimedia 2026

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code will be released soon.

Citations

• Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
• Aesthetics is Cheap, Show me the Text: An Empirical Evaluation of State-of-the-Art Generative Models for OCR
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• Visual-RFT: Visual Reinforcement Fine-Tuning
• Smaller But Better: Unifying Layout Generation with Smaller Large Language Models
• LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
• VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension
• Osprey: Pixel Understanding with Visual Instruction Tuning
• Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
• Kosmos-2: Grounding Multimodal Large Language Models to the World
• SPTS v2: Single-Point Scene Text Spotting
• DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting
• Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion
• ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
• ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network
• Scene Text Detection and Recognition: The Deep Learning Era
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes
• FOTS: Fast Oriented Text Spotting with a Unified Network
• Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition
• Rationalizing Neural Predictions
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Related

• Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs
• Grounding Everything in Tokens for Multimodal Large Language Models
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
• Visual Position Prompt for MLLM based Visual Grounding
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation