SAMPLE-HD: Simultaneous Action and Motion Planning Learning Environment

2022/06/01 by Michał Nazarczuk, Tony Ng, Nazarczuk, Michal +3
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Robotics (cs.RO) #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2206.10312

openalex publication_date 2022/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Humans exhibit incredibly high levels of multi-modal understanding - combining visual cues with read, or heard knowledge comes easy to us and allows for very accurate interaction with the surrounding environment. Various simulation environments focus on providing data for tasks related to scene understanding, question answering, space exploration, visual navigation. In this work, we are providing a solution to encompass both, visual and behavioural aspects of simulation in a new environment for learning interactive reasoning in manipulation setup. SAMPLE-HD environment allows to generate various scenes composed of small household objects, to procedurally generate language instructions for manipulation, and to generate ground truth paths serving as training data.

Related

• Learning multimodal representations for sample-efficient recognition of human actions
• Situational Grounding within Multimodal Simulations
• SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models
• Jointly Understand Your Command and Intention:Reciprocal Co-Evolution between Scene-Aware 3D Human Motion Synthesis and Analysis
• HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes