DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures

2025/07/11 by Uthayasooriyar, Benno, Ly, Antoine, Vermet, Franck +1
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.08606

Abstract

We introduce DocPolarBERT, a layout-aware BERT model for document understanding that eliminates the need for absolute 2D positional embeddings. We extend self-attention to take into account text block positions in relative polar coordinate system rather than the Cartesian one. Despite being pre-trained on a dataset more than six times smaller than the widely used IIT-CDIP corpus, DocPolarBERT achieves state-of-the-art results. These results demonstrate that a carefully designed attention mechanism can compensate for reduced pre-training data, offering an efficient and effective alternative for document understanding.

Citations

• Training LayoutLM from Scratch for Efficient Named-Entity Recognition in the Insurance Domain
• ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-training
• Length Generalization of Causal Transformers without Position Encoding
• LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding
• InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions
• DocLLM: A layout-aware generative language model for multimodal document understanding
• LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding
• ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation
• DocILE Benchmark for Document Information Localization and Extraction
• Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
• Structural generalization is hard for sequence-to-sequence models
• Unified Pretraining Framework for Document Understanding
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
• LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding
• BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents
• DocFormer: End-to-End Transformer for Document Understanding
• Going Full-TILT Boogie on Document Understanding with Text-Image-Layout\n Transformer
• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
• Span-based Semantic Parsing for Compositional Generalization
• Spatial Dependency Parsing for Semi-Structured Document Information Extraction
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
• Self-Attention with Relative Position Representations
• Attention Is All You Need
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Evaluation of Deep Convolutional Nets for Document Image Classification\n and Retrieval
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Skim-Attention: Learning to Focus via Document Layout
• Enhancing Visually-Rich Document Understanding via Layout Structure Modeling
• Going Full-TILT Boogie on Document Understanding with Text-Image-Layout\n Transformer