InspectVLM: Unified in Theory, Unreliable in Practice

2025/08/03 by Wallace, Conor, Corley, Isaac, Lwowski, Jonathan
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.01921

Abstract

Unified vision-language models (VLMs) promise to streamline computer vision pipelines by reframing multiple visual tasks such as classification, detection, and keypoint localization within a single language-driven interface. This architecture is particularly appealing in industrial inspection, where managing disjoint task-specific models introduces complexity, inefficiency, and maintenance overhead. In this paper, we critically evaluate the viability of this unified paradigm using InspectVLM, a Florence-2-based VLM trained on InspectMM, our new large-scale multimodal, multitask inspection dataset. While InspectVLM performs competitively on image-level classification and structured keypoint tasks, we find that it fails to match traditional ResNet-based models in core inspection metrics. Notably, the model exhibits brittle behavior under low prompt variability, produces degenerate outputs for fine-grained object detection, and frequently defaults to memorized language responses regardless of visual input. Our findings suggest that while language-driven unification offers conceptual elegance, current VLMs lack the visual grounding and robustness necessary for deployment in precision critical industrial inspections.

Citations

• Solar Panel Mapping via Oriented Object Detection
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• PaliGemma: A versatile 3B VLM for transfer
• Barely-Visible Surface Crack Detection for Wind Turbine Sustainability
• Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
• PaLI-3 Vision Language Models: Smaller, Faster, Stronger
• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Visual Instruction Tuning
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• DaViT: Dual Attention Vision Transformers
• Decoupled Weight Decay Regularization
• Mask R-CNN
• Deep Residual Learning for Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Related

• Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Model
• Hidden in plain sight: VLMs overlook their visual representations
• ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
• From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
• UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling