Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection

2024/01/11 by Jiang, Weibo, Ren, Weihong, Tian, Jiandong +3 · 3 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2401.05676

Abstract

Human-Object Interaction (HOI) detection plays a vital role in scene understanding, which aims to predict the HOI triplet in the form of . Existing methods mainly extract multi-modal features (e.g., appearance, object semantics, human pose) and then fuse them together to directly predict HOI triplets. However, most of these methods focus on seeking for self-triplet aggregation, but ignore the potential cross-triplet dependencies, resulting in ambiguity of action prediction. In this work, we propose to explore Self- and Cross-Triplet Correlations (SCTC) for HOI detection. Specifically, we regard each triplet proposal as a graph where Human, Object represent nodes and Action indicates edge, to aggregate self-triplet correlation. Also, we try to explore cross-triplet dependencies by jointly considering instance-level, semantic-level, and layout-level relations. Besides, we leverage the CLIP model to assist our SCTC obtain interaction-aware feature by knowledge distillation, which provides useful action clues for HOI detection. Extensive experiments on HICO-DET and V-COCO datasets verify the effectiveness of our proposed SCTC.

Cited by

• Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache
• Learning Human-Object Interaction as Groups
• Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration

Related

• Detecting Human-Object Interactions with Action Co-occurrence Priors
• Detecting and Recognizing Human-Object Interactions
• Learning Human-Object Interaction Detection using Interaction Points
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection
• Exploiting Scene Graphs for Human-Object Interaction Detection