Unified Human-Scene Interaction via Prompted Chain-of-Contacts

2023/09/14 by Zeqi Xiao, Tai Wang, Xiao, Zeqi +13 · 18 citations
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Motion and Animation #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2309.07918

openalex publication_date 2023/09/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausibility, two pivotal factors, versatile interaction control and the development of a user-friendly interface, require further exploration before the practical application of HSI. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands. This framework is built upon the definition of interaction as Chain of Contacts (CoC): steps of human joint-object part pairs, which is inspired by the strong correlation between interaction types and human-object contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution. To facilitate training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability to real scanned scenes. The project page is at https://github.com/OpenRobotLab/UniHSI .

Cited by

• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
• Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions
• Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contexts
• BRIC: Bridging Kinematic Plans and Physical Control at Test Time
• Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World
• MOGRAS: Human Motion with Grasping in 3D Scenes
• Advances in 4D Representation: Geometry, Motion, and Interaction
• H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows
• PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System
• Learning to Ball: Composing Policies for Long-Horizon Basketball Moves
• HumanoidVerse: A Versatile Humanoid for Vision-Language Guided Multi-Object Rearrangement
• Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generation
• DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Experts
• Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
• Half-Physics: Enabling Kinematic 3D Human Model with Physical Interactions
• Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data
• ULC: A Unified and Fine-Grained Controller for Humanoid Loco-Manipulation
• A Survey of Behavior Foundation Model: Next-Generation Whole-Body Control System of Humanoid Robots

Related

• CG-HOI: Contact-Guided 3D Human-Object Interaction Generation
• TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization
• InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation
• COUCH: Towards Controllable Human-Chair Interactions
• Uni-Inter: Unifying 3D Human Motion Synthesis Across Diverse Interaction Contexts