Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
2026/03/10 by Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4 · 1 voice
Computer Science · #cs.CL #cs.CV
paper · pdf
arxiv published 2026/03/10 · arxiv updated 2026/05/30
Abstract
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.
Citations
- VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
- Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
- ARC Is a Vision Problem!
- See the Text: From Tokenization to Visual Reading
- Glyph: Scaling Context Windows via Visual-Text Compression
- HICode: Hierarchical Inductive Coding with LLMs
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models
- SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
- ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
- Qwen3 Technical Report
- BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
- Gemma 3 Technical Report
- Why Do Multi-Agent LLM Systems Fail?
- PixelWorld: How Far Are We from Perceiving Everything as Pixels?
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- Pixtral 12B
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Language Modelling with Pixels
- Training Verifiers to Solve Math Word Problems
- Evaluating Large Language Models Trained on Code
- LoRA: Low-Rank Adaptation of Large Language Models
- A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers
- Measuring Massive Multitask Language Understanding
- Aligning AI With Shared Human Values
- TextCaps: a Dataset for Image Captioning with Reading Comprehension
- Computational Grounded Theory: A Methodological Framework
- Scene Text Visual Question Answering
- Towards VQA Models That Can Read
- Know What You Don't Know: Unanswerable Questions for SQuAD
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Cross Modal Distillation for Supervision Transfer
Discussions
Related