Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models

2025/05/14 by Lucas Choi, Ross Greer, Choi, Lucas +1
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.09139

Abstract

Vision-language models (VLMs) offer flexible object detection through natural language prompts but suffer from performance variability depending on prompt phrasing. In this paper, we introduce a method for automated prompt refinement using a novel metric called the Contrastive Class Alignment Score (CCAS), which ranks prompts based on their semantic alignment with a target object class while penalizing similarity to confounding classes. Our method generates diverse prompt candidates via a large language model and filters them through CCAS, computed using prompt embeddings from a sentence transformer. We evaluate our approach on challenging object categories, demonstrating that our automatic selection of high-precision prompts improves object detection accuracy without the need for additional model training or labeled data. This scalable and model-agnostic pipeline offers a principled alternative to manual prompt engineering for VLM-based detection systems.

Citations

• Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety
• Can Vision-Language Models Understand and Interpret Dynamic Gestures from Pedestrians? Pilot Datasets and Exploration Towards Instructive Nonverbal Commands for Cooperative Autonomous Vehicles
• CamoSAM2: Motion-Appearance Induced Auto-Refining Prompts for Video Camouflaged Object Detection
• GPT-4o System Card
• IPO: Interpretable Prompt Optimization for Vision-Language Models
• Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety
• Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmets
• African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification
• SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
• Driver Activity Classification Using Generalizable Representations from Vision-Language Models
• Language-Driven Active Learning for Diverse Open-Set 3D Object Detection
• T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy
• Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets
• Zero-shot Nuclei Detection via Visual-Language Pre-trained Models
• Scaling Open-Vocabulary Object Detection
• Fine-Grained Visual Prompting
• Micrograph segmentations for DDEVD
• Segment Anything
• Prompt Distribution Learning
• Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model
• Learning to Prompt for Vision-Language Models
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Generation and Comprehension of Unambiguous Object Descriptions

Related

• PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination
• Prompt-aligned Gradient for Prompt Tuning
• Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment
• Modeling Variants of Prompts for Vision-Language Models
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models