HOTR: End-to-End Human-Object Interaction Detection with Transformers

2021/04/28 by Bumsoo Kim, Junhyun Lee, Kim, Bumsoo +7 · 19 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2104.13682

openalex publication_date 2021/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred to by HOTR, which directly predicts a set of triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.

Citations

• Deep Contextual Attention for Human-Object Interaction Detection
• Visual Compositional Learning for Human-Object Interaction Detection

Cited by

• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
• Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache
• UniHOI: Unified Human-Object Interaction Understanding via Unified Token Space
• Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection
• Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection
• Learning Human-Object Interaction as Groups
• HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection
• Contextualized Representation Learning for Effective Human-Object Interaction Detection
• Pointing-Guided Target Estimation via Transformer-Based Attention
• DQEN: Dual Query Enhancement Network for DETR-based HOI Detection
• Rethinking Human-Object Interaction Evaluation for both Vision-Language Models and HOI-Specific Methods
• EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba
• QueryCraft: Transformer-Guided Query Initialization for Enhanced Human-Object Interaction Detection
• Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
• RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection
• A document is worth a structured record: Principled inductive bias design for document recognition
• Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection
• Egocentric Human-Object Interaction Detection: A New Benchmark and Method

Related

• Human-Object Interaction Detection via Disentangled Transformer
• End-to-End Human Object Interaction Detection with HOI Transformer
• MSTR: Multi-Scale Transformer for End-to-End Human-Object Interaction Detection
• What to look at and where: Semantic and Spatial Refined Transformer for detecting human-object interactions
• Efficient Human-Object-Interaction (EHOI) Detection via Interaction Label Coding and Conditional Decision