LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models

2024/07/27 by Ruiyi Zhang, Zhang, Ruiyi, Yufan Zhou +9 · 6 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2407.19185

openalex publication_date 2024/07/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large multimodal language models have demonstrated impressive capabilities in understanding and manipulating images. However, many of these models struggle with comprehending intensive textual contents embedded within the images, primarily due to the limited text recognition and layout understanding ability. To understand the sources of these limitations, we perform an exploratory analysis showing the drawbacks of classical visual encoders on visual text understanding. Hence, we present LLaVA-Read, a multimodal large language model that utilizes dual visual encoders along with a visual text encoder. Our model surpasses existing state-of-the-art models in various text-rich image understanding tasks, showcasing enhanced comprehension of textual content within images. Together, our research suggests visual text understanding remains an open challenge and an efficient visual text encoder is crucial for future successful multimodal systems.

Cited by

• Zenbo Patrol: A Social Assistive Robot Based on Multimodal Deep Learning for Real-time Illegal Parking Recognition and Notification
• SynDoc: A Hybrid Discriminative-Generative Framework for Enhancing Synthetic Domain-Adaptive Document Key Information Extraction
• KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts
• A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends
• MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models
• Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding

Related

• Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
• Improving Multi-modal Large Language Model through Boosting Vision Capabilities
• ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
• LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
• Rethinking Visual Information Processing in Multimodal LLMs