Evaluating Graphical Perception with Multimodal LLMs

2025/04/05 by Nguyen, Rami Huu, K. Maeda, Mahsa Geshvadi +4
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Data Visualization and Analytics #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2504.04221

openalex publication_date 2025/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) have remarkably progressed in analyzing and understanding images. Despite these advancements, accurately regressing values in charts remains an underexplored area for MLLMs. For visualization, how do MLLMs perform when applied to graphical perception tasks? Our paper investigates this question by reproducing Cleveland and McGill's seminal 1984 experiment and comparing it against human task performance. Our study primarily evaluates fine-tuned and pretrained models and zero-shot prompting to determine if they closely match human graphical perception. Our findings highlight that MLLMs outperform human task performance in some cases but not in others. We highlight the results of all experiments to foster an understanding of where MLLMs succeed and fail when applied to data visualization.

Citations

• Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models
• Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
• M²PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning
• MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
• Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
• Multimodal Reasoning with Multimodal Knowledge Graph
• Exploring Perceptual Limitation of Multimodal Large Language Models
• ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning
• V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
• ChartLlama: A Multimodal LLM for Chart Understanding and Generation
• MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
• What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
• KOSMOS-2.5: A Multimodal Literate Model
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Towards AGI in Computer Vision: Lessons Learned from GPT and Large Language Models
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
• Visual Instruction Tuning
• DVQA: Understanding Data Visualizations via Question Answering
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• VQA: Visual Question Answering

Related

• Towards Understanding Graphical Perception in Large Multimodal Models
• The Plot Thickens: Quantitative Part-by-Part Exploration of MLLM Visualization Literacy
• Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
• Understanding Graphical Perception in Data Visualization through Zero-shot Prompting of Vision-Language Models
• Visualization Literacy of Multimodal Large Language Models: A Comparative Study