2024/03/04 by Qi Liu, Yuxiao Wang, Liu, Qi +11 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Semantic Web and Ontologies #Web Data Mining and Analysis
paper · pdf · doi:10.48550/arxiv.2403.01840
openalex publication_date 2024/03/04 · openalex created_date 2024/03/06 · openalex updated_date 2026/07/28
Recent human-object interaction (HOI) detection methods depend on extensively annotated image datasets, which require a significant amount of manpower. In this paper, we propose a novel self-adaptive, language-driven HOI detection method, termed FreeA. This method leverages the adaptability of the text-image model to generate latent HOI labels without requiring manual annotation. Specifically, FreeA aligns image features of human-object pairs with HOI text templates and employs a knowledge-based masking technique to decrease improbable interactions. Furthermore, FreeA implements a proposed method for matching interaction correlations to increase the probability of actions associated with a particular action, thereby improving the generated HOI labels. Experiments on two benchmark datasets showcase that FreeA achieves state-of-the-art performance among weakly supervised HOI competitors. Our proposal gets +13.29 (159%\uparrow) mAP and +17.30 (98%\uparrow) mAP than the newest ``Weakly'' supervised model, and +7.19 (28%\uparrow) mAP and +14.69 (34%\uparrow) mAP than the latest ``Weakly+'' supervised model, respectively, on HICO-DET and V-COCO datasets, more accurate in localizing and classifying the interactive actions. The source code will be made public.