Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts

2024/10/16 by Sri Harsha Dumpala, Aman Jaiswal, Dumpala, Sri Harsha +9 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2410.13030

openalex publication_date 2024/10/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs.

Cited by

• Promptception: How Sensitive Are Large Multimodal Models to Prompts?

Related

• SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
• PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
• Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
• VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
• V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models