An End-to-End Human Simulator for Task-Oriented Multimodal Human-Robot Collaboration

2023/04/02 by Shervedani, Afagh Mehri, Li, Siyu, Monaikul, Natawut +3
#FOS: Computer and information sciences #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2304.00584

Abstract

This paper proposes a neural network-based user simulator that can provide a multimodal interactive environment for training Reinforcement Learning (RL) agents in collaborative tasks involving multiple modes of communication. The simulator is trained on the existing ELDERLY-AT-HOME corpus and accommodates multiple modalities such as language, pointing gestures, and haptic-ostensive actions. The paper also presents a novel multimodal data augmentation approach, which addresses the challenge of using a limited dataset due to the expensive and time-consuming nature of collecting human demonstrations. Overall, the study highlights the potential for using RL and multimodal user simulators in developing and improving domestic assistive robots.

Related

• Multimodal Reinforcement Learning for Robots Collaborating with Humans
• Assistax: A Multi-Agent Hardware-Accelerated Reinforcement Learning Benchmark for Assistive Robotics
• Evaluating Multimodal Interaction of Robots Assisting Older Adults
• ARROCH: Augmented Reality for Robots Collaborating with a Human
• MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication