Handwritten Text Recognition of Historical Manuscripts Using Transformer-Based Models

2025/08/15 by Meoded, Erez
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2508.11499

Abstract

Historical handwritten text recognition (HTR) is essential for unlocking the cultural and scholarly value of archival documents, yet digitization is often hindered by scarce transcriptions, linguistic variation, and highly diverse handwriting styles. In this study, we apply TrOCR, a state-of-the-art transformer-based HTR model, to 16th-century Latin manuscripts authored by Rudolf Gwalther. We investigate targeted image preprocessing and a broad suite of data augmentation techniques, introducing four novel augmentation methods designed specifically for historical handwriting characteristics. We also evaluate ensemble learning approaches to leverage the complementary strengths of augmentation-trained models. On the Gwalther dataset, our best single-model augmentation (Elastic) achieves a Character Error Rate (CER) of 1.86, while a top-5 voting ensemble achieves a CER of 1.60 - representing a 50% relative improvement over the best reported TrOCR_BASE result and a 42% improvement over the previous state of the art. These results highlight the impact of domain-specific augmentations and ensemble strategies in advancing HTR performance for historical manuscripts.

Citations

• Best Practices for a Handwritten Text Recognition System
• On fine-tuning deep learning models using transfer learning and hyper-parameters optimization for disease identification in maize leaves
• Transformer-based HTR for Historical Documents
• You Only Cut Once: Boosting Data Augmentation with a Single Cut
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Rethinking the Hyperparameters for Fine-tuning
• Unsupervised Cross-lingual Representation Learning at Scale
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Attention Is All You Need
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Neural Machine Translation by Jointly Learning to Align and Translate
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Open Source Handwritten Text Recognition on Medieval Manuscripts using Mixed Models and Document-Specific Finetuning
• How to Choose Pretrained Handwriting Recognition Models for Single Writer Fine-Tuning
• Application of deep learning approaches for medieval historical documents transcription
• The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition
• Unlocking the archives: Using large language models to transcribe handwritten historical documents