The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models

2023/10/04 by Chenwei Wu, Wu, Chenwei, Li Erran Li +9 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.02777

openalex publication_date 2023/10/04 · openalex created_date 2023/10/06 · openalex updated_date 2026/07/28

Abstract

Compositionality is a common property in many modalities including natural languages and images, but the compositional generalization of multi-modal models is not well-understood. In this paper, we identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We show that current attempts to improve compositional generalization rely on linguistic priors rather than on information in the image. We also propose a new metric for compositionality without such linguistic priors.

Cited by

• A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs

Related

• Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
• Causal Graphical Models for Vision-Language Compositional Understanding
• MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
• Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View