VISaGE: Understanding Visual Generics and Exceptions

2025/10/14 by Stella Frank, Frank, Stella, Emily Allaway +1
Computer Science · Psychology · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Language, Metaphor, and Cognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2510.12548

openalex publication_date 2025/10/14 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

While Vision Language Models (VLMs) learn conceptual representations, in the form of generalized knowledge, during training, they are typically used to analyze individual instances. When evaluation instances are atypical, this paradigm results in tension between two priors in the model. The first is a pragmatic prior that the textual and visual input are both relevant, arising from VLM finetuning on congruent inputs; the second is a semantic prior that the conceptual representation is generally true for instances of the category. In order to understand how VLMs trade off these priors, we introduce a new evaluation dataset, VISaGE, consisting of both typical and exceptional images. In carefully balanced experiments, we show that conceptual understanding degrades when the assumption of congruency underlying the pragmatic prior is violated with incongruent images. This effect is stronger than the effect of the semantic prior when querying about individual instances.

Citations

• Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era
• Probing Visual Language Priors in VLMs
• NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
• Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?
• IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
• Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
• Training language models to follow instructions with human feedback
• Perceptual Score: What Data Modalities Does Your Model Perceive?
• Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
• A Unified Approach to Interpreting Model Predictions
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Cited by

• Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences

Related

• If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions
• CIVET: Systematic Evaluation of Understanding in VLMs
• Would you still call this Dax? Novel Visual References in VLMs and Humans
• SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models
• Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features