THOR: Text to Human-Object Interaction Diffusion via Relation Intervention

2024/03/17 by Qianyang Wu, Wu, Qianyang, Shi Ye +9 · 7 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2403.11208

openalex publication_date 2024/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper addresses new methodologies to deal with the challenging task of generating dynamic Human-Object Interactions from textual descriptions (Text2HOI). While most existing works assume interactions with limited body parts or static objects, our task involves addressing the variation in human motion, the diversity of object shapes, and the semantic vagueness of object motion simultaneously. To tackle this, we propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR). THOR is a cohesive diffusion model equipped with a relation intervention mechanism. In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion. This intervention enhances the spatial-temporal relations between humans and objects, with human-centric interaction representation providing additional guidance for synthesizing consistent motion from text. To achieve more reasonable and realistic results, interaction losses is introduced at different levels of motion granularity. Moreover, we construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset. Both quantitative and qualitative experiments demonstrate the effectiveness of our proposed model.

Cited by

• Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contexts
• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation
• SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learning
• Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis
• Occlusion-Aware Temporally Consistent Amodal Completion for 3D Human-Object Interaction Reconstruction
• HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion
• GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects

Related

• HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models
• CG-HOI: Contact-Guided 3D Human-Object Interaction Generation
• InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
• Learning to Generate Human-Human-Object Interactions from Textual Descriptions
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models