Focus Anywhere for Fine-grained Multi-page Document Understanding

2024/05/23 by Chenglong Liu, Haoran Wei, Liu, Chenglong +17 · 11 citations
Arts and Humanities · Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Digital Humanities and Scholarship #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2405.14295

openalex publication_date 2024/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Modern LVLMs still struggle to achieve fine-grained document understanding, such as OCR/translation/caption for regions of interest to the user, tasks that require the context of the entire page, or even multiple pages. Accordingly, this paper proposes Fox, an effective pipeline, hybrid data, and tuning strategy, that catalyzes LVLMs to focus anywhere on single/multi-page documents. We introduce a novel task to boost the document understanding by making LVLMs focus attention on the document-level region, such as redefining full-page OCR as foreground focus. We employ multiple vision vocabularies to extract visual hybrid knowledge for interleaved document pages (e.g., a page containing a photo). Meanwhile, we render cross-vocabulary vision data as the catalyzer to achieve a full reaction of multiple visual vocabularies and in-document figure understanding. Further, without modifying the weights of multiple vision vocabularies, the above catalyzed fine-grained understanding capabilities can be efficiently tuned to multi-page documents, enabling the model to focus anywhere in both format-free and page-free manners. Besides, we build a benchmark including 9 fine-grained sub-tasks (e.g., region-level OCR/summary, color-guided OCR) to promote document analysis in the community. The experimental results verify the superiority of our model.

Cited by

• DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation
• Context Cascade Compression: Exploring the Upper Limits of Text Compression
• UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models
• DeepSeek-OCR: Contexts Optical Compression
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
• MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs
• PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models
• Large Language Models in Document Intelligence: A Comprehensive Survey, Recent Advances, Challenges, and Future Trends
• POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

Related

• A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends
• MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
• VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding
• DocVLM: Make Your VLM an Efficient Reader
• Doc-CoB: Enhancing Multi-Modal Document Understanding with Visual Chain-of-Boxes Reasoning