Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers

2025/07/14 by Yilun Zhao, Chengye Wang, Zhao, Yilun +5 · 1 citation
Computer Science · #Advanced Text Analysis Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.10787

openalex publication_date 2025/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper introduces MISS-QA, the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature. MISS-QA comprises 1,500 expert-annotated examples over 465 scientific papers. In this benchmark, models are tasked with interpreting schematic diagrams that illustrate research overviews and answering corresponding information-seeking questions based on the broader context of the paper. We assess the performance of 18 frontier multimodal foundation models, including o4-mini, Gemini-2.5-Flash, and Qwen2.5-VL. We reveal a significant performance gap between these models and human experts on MISS-QA. Our analysis of model performance on unanswerable questions and our detailed error analysis further highlight the strengths and limitations of current models, offering key insights to enhance models in comprehending multimodal scientific literature.

Citations

• SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
• Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediation
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers
• Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Know What You Don't Know: Unanswerable Questions for SQuAD
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• SQuAD: 100,000+ Questions for Machine Comprehension of Text

Cited by

• AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

Related

• M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models
• SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification
• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
• SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials
• MMCR: Benchmarking Cross-Source Reasoning in Scientific Papers