Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models

2023/09/07 by Jiaying Lu, Jinmeng Rao, Lu, Jiaying +13 · 1 voice · 3 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.2309.04041

openalex publication_date 2023/09/07 · arxiv published 2023/09/07 · arxiv updated 2024/01/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models (LVLMs) offer remarkable benefits for a variety of vision-language tasks. However, a challenge hindering their application in real-world scenarios, particularly regarding safety, robustness, and reliability, is their constrained semantic grounding ability, which pertains to connecting language to the physical-world entities or concepts referenced in images. Therefore, a crucial need arises for a comprehensive study to assess the semantic grounding ability of widely used LVLMs. Despite the significance, sufficient investigation in this direction is currently lacking. Our work bridges this gap by designing a pipeline for generating large-scale evaluation datasets covering fine-grained semantic information, such as color, number, material, etc., along with a thorough assessment of seven popular LVLMs' semantic grounding ability. Results highlight prevalent misgrounding across various aspects and degrees. To address this issue, we propose a data-centric enhancement method that aims to improve LVLMs' semantic grounding ability through multimodal instruction tuning on fine-grained conversations. Experiments on enhanced LVLMs demonstrate notable improvements in addressing misgrounding issues.

Cited by

• Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
• Agentic Design Review System
• PictSure: Pretraining Embeddings Matters for In-Context Learning Image Classifiers

Discussions

• 📝 Evaluation and Mitigation of Agnosia in Multimodal Large Language Models 🔭📚 "Proposes EMMA, an evaluation-mitigation framework that automatically creates fine-grained and diverse visual question [mastodon, 0 points, 0 comments]

Related

• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
• Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?
• Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models
• Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding