Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation

2025/04/22 by Ziqiao Ma, Ma, Ziqiao, Jing Ding +13 · 4 citations
Computer Science · Psychology · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language, Metaphor, and Cognition #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2504.16060

openalex publication_date 2025/04/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative communication (Grice, 1975). However, current evaluations of vision-language models (VLMs) often overlook the pragmatic dimension, reducing REG to a region-based captioning task and neglecting Gricean maxims. In this work, we revisit REG from a pragmatic perspective, introducing a new dataset (RefOI) of 1.5k images annotated with both written and spoken referring expressions. Through a systematic evaluation of state-of-the-art VLMs, we identify three key failures of pragmatic competence: (1) failure to uniquely identify the referent, (2) inclusion of excessive or irrelevant information, and (3) misalignment with human pragmatic preference, such as the underuse of minimal spatial cues. We also show that standard automatic evaluations fail to capture these pragmatic violations, reinforcing superficial cues rather than genuine referential success. Our findings call for a renewed focus on pragmatically informed models and evaluation frameworks that align with real human communication.

Citations

• VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation
• Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges
• Logical forms complement probability in understanding language model (and human) performance
• Exploring Spatial Language Grounding Through Referring Expressions
• Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities
• Grounding Language in Multi-Perspective Referential Communication
• Benchmarks as Microscopes: A Call for Model Metrology
• Multi-Object Hallucination in Vision-Language Models
• Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models
• Learning Language Structures through Grounding
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
• GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
• Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
• GSVA: Generalized Segmentation via Multimodal Large Language Models
• VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• CogVLM: Visual Expert for Pretrained Language Models
• GLaMM: Pixel Grounding Large Multimodal Model
• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
• Ferret: Refer and Ground Anything Anywhere at Any Granularity
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
• LISA: Reasoning Segmentation via Large Language Model
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models
• Fine-Grained Visual Prompting
• GRES: Generalized Referring Expression Segmentation
• DisCLIP: Open-Vocabulary Referring Expression Generation
• Gender Biases in Automatic Evaluation Metrics for Image Captioning
• Visual Instruction Tuning
• What does CLIP know about a red circle? Visual prompt engineering for VLMs
• The Vector Grounding Problem
• Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models
• A fine-grained comparison of pragmatic language understanding in humans and language models
• InstructPix2Pix: Learning to Follow Image Editing Instructions
• Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
• Word Discovery in Visually Grounded, Self-Supervised Speech Models
• Learning to Mediate Disparities Towards Pragmatic Communication
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Grounded Language-Image Pre-training
• Few-shot Language Coordination by Modeling Theory of Mind
• MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning
• AmbigQA: Answering Ambiguous Open-domain Questions
• UNITER: UNiversal Image-TExt Representation Learning
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• BERTScore: Evaluating Text Generation with BERT
• Large-scale interactive object segmentation with human annotators
• Modeling Context in Referring Expressions
• SPICE: Semantic Propositional Image Caption Evaluation
• Generation and Comprehension of Unambiguous Object Descriptions
• CIDEr: Consensus-based Image Description Evaluation
• Microsoft COCO: Common Objects in Context
• Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions
• VAGUE: Visual Contexts Clarify Ambiguous Expressions
• Understanding natural language

Cited by

• Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models
• On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts

Related

• Comprehension-guided referring expressions
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Modeling Context in Referring Expressions
• RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
• Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions