Document Understanding Dataset and Evaluation (DUDE)

2023/05/15 by Jordy Van Landeghem, Van Landeghem, Jordy, Rubèn Tito +23 · 29 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.08455

openalex publication_date 2023/05/15 · openalex created_date 2023/05/17 · openalex updated_date 2026/07/28

Abstract

We call on the Document AI (DocAI) community to reevaluate current methodologies and embrace the challenge of creating more practically-oriented benchmarks. Document Understanding Dataset and Evaluation (DUDE) seeks to remediate the halted research progress in understanding visually-rich documents (VRDs). We present a new dataset with novelties related to types of questions, answers, and document layouts based on multi-industry, multi-domain, and multi-page VRDs of various origins, and dates. Moreover, we are pushing the boundaries of current methods by creating multi-task and multi-domain evaluation setups that more accurately simulate real-world situations where powerful generalization and adaptation under low-resource settings are desired. DUDE aims to set a new standard as a more practical, long-standing benchmark for the community, and we hope that it will lead to future extensions and contributions that address real-world challenges. Finally, our work illustrates the importance of finding more efficient ways to model language, images, and layout in DocAI.

Cited by

• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
• Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark
• HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding
• Unexplored flaws in multiple-choice VQA evaluations
• HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documents
• Arctic-Extract Technical Report
• VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
• SciEGQA: A Dataset for Scientific Evidence-Grounded Question Answering and Reasoning
• FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR Evaluation
• DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
• ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents
• Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracy
• SCoPE VLM: Selective Context Processing for Efficient Document Navigation in Vision-Language Models
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding
• Document Intelligence in the Era of Large Language Models: A Survey
• LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding
• Towards Reliable and Interpretable Document Question Answering via VLMs
• Hierarchical Vision-Language Reasoning for Multimodal Multiple-Choice Question Answering
• Enhancing Document VQA Models via Retrieval-Augmented Generation
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding
• DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understanding
• Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
• MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
• Multi-Agent Interactive Question Generation Framework for Long Document Understanding
• MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Tasks
• Docopilot: Improving Multimodal Models for Document-Level Understanding
• Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark

Related

• Document AI: Benchmarks, Models and Applications
• BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations
• BuDDIE: A Business Document Dataset for Multi-task Information Extraction
• DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception
• DocBank: A Benchmark Dataset for Document Layout Analysis