MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling

2025/06/12 by Liang Yin, Yin, Liang, Xudong Xie +7
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2506.10609

openalex publication_date 2025/06/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Scene text retrieval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Muti-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and 16k images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5%. The code and datasets are available at https://github.com/yingift/MSTAR.

Citations

• Capturing More: Learning Multi-Domain Representations for Robust Online Handwriting Verification
• Privacy-Preserving Biometric Verification with Handwritten Random Digit String
• Partial Scene Text Retrieval
• PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling
• Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
• WeCromCL: Weakly Supervised Cross-Modality Contrastive Learning for Transcription-only Supervised Text Spotting
• ColPali: Efficient Document Retrieval with Vision Language Models
• Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
• Bridging the Gap Between End-to-End and Two-Step Text Spotting
• Fine-Grained Visual Prompting
• ICDAR 2023 Competition on Hierarchical Text Detection and Recognition
• What does CLIP know about a red circle? Visual prompt engineering for VLMs
• Sigmoid Loss for Language Image Pre-Training
• Turning a CLIP Model into a Scene Text Detector
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• CLIPTER: Looking at the Bigger Picture in Scene Text Recognition
• SPTS v2: Single-Point Scene Text Spotting
• DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting
• Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition
• ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval
• Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• SPTS: Single-Point Text Spotting
• Masked-attention Mask Transformer for Universal Image Segmentation
• ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
• Scene Text Retrieval via Joint Text Detection and Similarity Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting
• Structured Query-Based Image Retrieval Using Scene Graphs
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• TextCaps: a Dataset for Image Captioning with Reading Comprehension
• ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection\n and Recognition -- RRC-MLT-2019
• Single Shot Scene Text Retrieval
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
• Synthetic Data for Text Localisation in Natural Images
• COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
• Reading Text in the Wild with Convolutional Neural Networks

Related

• Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval
• Scene Text Retrieval via Joint Text Detection and Similarity Learning
• Partial Scene Text Retrieval
• Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
• Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrieval