Hyper-Local Deformable Transformers for Text Spotting on Historical Maps

2025/06/17 by Yijun Lin, Lin, Yijun, Yao‐Yi Chiang +1 · 3 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Image Processing and 3D Reconstruction #Image Retrieval and Classification Techniques

paper · pdf · doi:10.48550/arxiv.2506.15010

openalex publication_date 2025/06/17 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Text on historical maps contains valuable information providing georeferenced historical, political, and cultural contexts. However, text extraction from historical maps is challenging due to the lack of (1) effective methods and (2) training data. Previous approaches use ad-hoc steps tailored to only specific map styles. Recent machine learning-based text spotters (e.g., for scene images) have the potential to solve these challenges because of their flexibility in supporting various types of text instances. However, these methods remain challenges in extracting precise image features for predicting every sub-component (boundary points and characters) in a text instance. This is critical because map text can be lengthy and highly rotated with complex backgrounds, posing difficulties in detecting relevant image features from a rough text region. This paper proposes PALETTE, an end-to-end text spotter for scanned historical maps of a wide variety. PALETTE introduces a novel hyper-local sampling module to explicitly learn localized image features around the target boundary points and characters of a text instance for detection and recognition. PALETTE also enables hyper-local positional embeddings to learn spatial interactions between boundary points and characters within and across text instances. In addition, this paper presents a novel approach to automatically generate synthetic map images, SynthMap+, for training text spotters for historical maps. The experiment shows that PALETTE with SynthMap+ outperforms SOTA text spotters on two new benchmark datasets of historical maps, particularly for long and angled text. We have deployed PALETTE with SynthMap+ to process over 60,000 maps in the David Rumsey Historical Map collection and generated over 100 million text labels to support map searching. The project is released at https://github.com/kartta-foundation/mapkurator-palette-doc.

Citations

• ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformer
• The mapKurator System: A Complete Pipeline for Extracting and Linking Text from Historical Maps
• Towards Unified Scene Text Spotting based on Sequence Generation
• DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting
• GLASS: Global to Local Attention for Scene-Text Spotting
• SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition
• DEER: Detection-agnostic End-to-End Recognizer for Scene Text Spotting
• Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
• PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network
• Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting
• An Automatic Approach for Generating Rich, Linked Geo-Metadata from Historical Map Images
• End-to-End Object Detection with Transformers
• ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network
• Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting
• All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting
• Convolutional Character Networks
• FOTS: Fast Oriented Text Spotting with a Unified Network
• Mask R-CNN
• Synthetic Data for Text Localisation in Natural Images
• Deep Residual Learning for Image Recognition
• An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition
• Synthetic Data and Artificial Neural Networks for Natural Scene Text\n Recognition

Cited by

• MapSAM2: Adapting SAM2 for Automatic Segmentation of Historical Map Images and Time Series
• Detecting Legend Items on Historical Maps Using GPT-4o with In-Context Learning
• LIGHT: Multi-Modal Text Linking on Historical Maps

Related

• The mapKurator System: A Complete Pipeline for Extracting and Linking Text from Historical Maps
• Leveraging LLMs and attention-mechanism for automatic annotation of historical maps
• An Automatic Approach for Generating Rich, Linked Geo-Metadata from Historical Map Images
• MapSAM: Adapting Segment Anything Model for Automated Feature Detection in Historical Maps
• LIGHT: Multi-Modal Text Linking on Historical Maps