Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection

2026/04/30 by Weihao Cao, Runqi Wang, Xiaoyue Duan +3
Computer Science · #cs.CV

paper · pdf · doi:10.48550/arxiv.2604.04444

Accepted to CVPR 2026

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.

Citations

• MR-GDINO: Efficient Open-World Continual Object Detection
• OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
• Zero-shot Generalizable Incremental Learning for Vision-Language Object Detection
• YOLO-World: Real-Time Open-Vocabulary Object Detection
• AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learning
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
• Conditional Prompt Learning for Vision-Language Models
• DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
• DN-DETR: Accelerate DETR Training by Introducing Query DeNoising
• RegionCLIP: Region-based Language-Image Pretraining
• Learning to Prompt for Continual Learning
• Grounded Language-Image Pre-training
• Learning to Prompt for Vision-Language Models
• MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
• Learning Transferable Visual Models From Natural Language Supervision
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark
• Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression
• Zero-Shot Learning -- The Good, the Bad and the Ugly
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
• Deep Residual Learning for Image Recognition
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Microsoft COCO: Common Objects in Context
• The ubiquitous Kronecker product
• A Stochastic Approximation Method
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding