Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection

2024/08/05 by Ting Lei, Lei, Ting, Shaofeng Yin +5 · 15 citations
Computer Science · Engineering · Physics and Astronomy · #Anomaly Detection Techniques and Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Infrared Target Detection Methodologies #Radiation Detection and Scintillator Technologies

paper · pdf · doi:10.48550/arxiv.2408.02484

openalex publication_date 2024/08/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Zero-shot Human-Object Interaction (HOI) detection has emerged as a frontier topic due to its capability to detect HOIs beyond a predefined set of categories. This task entails not only identifying the interactiveness of human-object pairs and localizing them but also recognizing both seen and unseen interaction categories. In this paper, we introduce a novel framework for zero-shot HOI detection using Conditional Multi-Modal Prompts, namely CMMP. This approach enhances the generalization of large foundation models, such as CLIP, when fine-tuned for HOI detection. Unlike traditional prompt-learning methods, we propose learning decoupled vision and language prompts for interactiveness-aware visual feature extraction and generalizable interaction classification, respectively. Specifically, we integrate prior knowledge of different granularity into conditional vision prompts, including an input-conditioned instance prior and a global spatial pattern prior. The former encourages the image encoder to treat instances belonging to seen or potentially unseen HOI concepts equally while the latter provides representative plausible spatial configuration of the human and object under interaction. Besides, we employ language-aware prompt learning with a consistency constraint to preserve the knowledge of the large foundation model to enable better generalization in the text branch. Extensive experiments demonstrate the efficacy of our detector with conditional multi-modal prompts, outperforming previous state-of-the-art on unseen classes of various zero-shot settings. The code and models are available at \urlhttps://github.com/ltttpku/CMMP.

Cited by

• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
• Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
• Learning Human-Object Interaction as Groups
• Contextualized Representation Learning for Effective Human-Object Interaction Detection
• Interact-Custom: Customized Human Object Interaction Image Generation
• DQEN: Dual Query Enhancement Network for DETR-based HOI Detection
• CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods
• AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning
• TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring
• Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
• HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation
• Improving Personalized Image Generation through Social Context Feedback
• Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
• Locality-Aware Zero-Shot Human-Object Interaction Detection
• Generalized Visual Relation Detection with Diffusion Models

Related

• End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation
• Open-World Human-Object Interaction Detection via Multi-modal Prompts
• Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
• Locality-Aware Zero-Shot Human-Object Interaction Detection
• Funnel-HOI: Top-Down Perception for Zero-Shot HOI Detection