Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models

2025/05/26 by Chae, Hyunsik, Seungwoo Yoon, Jaden Park +11 · 2 citations
Engineering · Materials Science · #Artificial Intelligence (cs.AI) #Categorization #Comprehension #Computer Vision and Pattern Recognition (cs.CV) #Domain (mathematical analysis) #FOS: Computer and information sciences #Focus (optics) #Machine Learning in Materials Science #Perception #Robotics and Automated Systems #Visual language #Visual perception #Visual reasoning

paper · pdf · doi:10.48550/arxiv.2505.20021

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Recent Vision-Language Models (VLMs) have demonstrated impressive multimodal comprehension and reasoning capabilities, yet they often struggle with trivially simple visual tasks. In this work, we focus on the domain of basic 2D Euclidean geometry and systematically categorize the fundamental, indivisible visual perception skills, which we refer to as atomic visual skills. We then introduce the Atomic Visual Skills Dataset (AVSD) for evaluating VLMs on the atomic visual skills. Using AVSD, we benchmark state-of-the-art VLMs and find that they struggle with these tasks, despite being trivial for adult humans. Our findings highlight the need for purpose-built datasets to train and evaluate VLMs on atomic, rather than composite, visual perception tasks.

Citations

• GPT-4o System Card
• Can Models Learn Skill Composition from Examples?
• Out-of-distribution generalization via composition: a lens through induction heads in Transformers
• Out-of-distribution generalization via composition: A lens through induction heads in Transformers
• LLaVA-OneVision: Easy Visual Task Transfer
• Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
• Multimodal Table Understanding
• Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
• An Introduction to Vision-Language Modeling
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• BLINK: Multimodal Large Language Models Can See but Not Perceive
• Reverse Training to Nurse the Reversal Curse
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• Dual Operating Modes of In-Context Learning
• Measuring Vision-Language STEM Skills of Neural Models
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning
• Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks
• Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Physics of Language Models: Part 3.2, Knowledge Manipulation
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• A Theory for Emergence of Complex Skills in Language Models
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
• Teaching Arithmetic to Small Transformers
• SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
• Revisiting the Role of Language Priors in Vision-Language Models
• How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
• Visual Instruction Tuning
• GPT-4 Technical Report
• Teaching CLIP to Count to Ten
• CREPE: Can Vision-Language Foundation Models Reason Compositionally?
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
• Measuring and Narrowing the Compositionality Gap in Language Models
• When and why vision-language models behave like bags-of-words, and what to do about it?
• VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations
• Large Language Models are Zero-Shot Reasoners
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Program Synthesis with Large Language Models
• PlotQA: Reasoning over Scientific Plots
• VizWiz Grand Challenge: Answering Visual Questions from Blind People
• DVQA: Understanding Data Visualizations via Question Answering
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• A Diagram Is Worth A Dozen Images
• VQA: Visual Question Answering

Cited by

• EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts
• AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models
• COMPACT: COMPositional Atomic-to-Complex Visual Capability Tuning

Related

• Visual Language Models show widespread visual deficits on neuropsychological tests
• Vision language models are unreliable at trivial spatial cognition
• Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding
• Decoupling the components of geometric understanding in Vision Language Models
• Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models