HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models

2023/03/28 by Shan Ning, Longtian Qiu, Ning, Shan +5 · 15 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2303.15786

openalex publication_date 2023/03/28 · openalex created_date 2023/04/05 · openalex updated_date 2026/07/28

Abstract

Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Recently, Contrastive Language-Image Pre-training (CLIP) has shown great potential in providing interaction prior for HOI detectors via knowledge distillation. However, such approaches often rely on large-scale training data and suffer from inferior performance under few/zero-shot scenarios. In this paper, we propose a novel HOI detection framework that efficiently extracts prior knowledge from CLIP and achieves better generalization. In detail, we first introduce a novel interaction decoder to extract informative regions in the visual feature map of CLIP via a cross-attention mechanism, which is then fused with the detection backbone by a knowledge integration block for more accurate human-object pair detection. In addition, prior knowledge in CLIP text encoder is leveraged to generate a classifier by embedding HOI descriptions. To distinguish fine-grained interactions, we build a verb classifier from training data via visual semantic arithmetic and a lightweight verb representation adapter. Furthermore, we propose a training-free enhancement to exploit global HOI predictions from CLIP. Extensive experiments demonstrate that our method outperforms the state of the art by a large margin on various settings, e.g. +4.04 mAP on HICO-Det. The source code is available in https://github.com/Artanic30/HOICLIP.

Cited by

• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
• VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification
• Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache
• Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning
• Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
• Learning Human-Object Interaction as Groups
• HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection
• Trainee Action Recognition through Interaction Analysis in CCATT Mixed-Reality Training
• Contextualized Representation Learning for Effective Human-Object Interaction Detection
• Multimodal Graph Network Modeling for Human-Object Interaction Detection with PDE Graph Diffusion
• No More Sibling Rivalry: Debiasing Human-Object Interaction Detection
• DQEN: Dual Query Enhancement Network for DETR-based HOI Detection
• Rethinking Human-Object Interaction Evaluation for both Vision-Language Models and HOI-Specific Methods
• Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration

Related

• Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model
• Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels
• Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration
• Disentangled Pre-training for Human-Object Interaction Detection
• Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning