Text2Action: Generative Adversarial Synthesis from Language to Action

2017/10/15 by Hyemin Ahn, Timothy Ha, Ahn, Hyemin +7 · 14 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO) #cs.CL #cs.LG #cs.RO

paper · pdf · doi:10.48550/arxiv.1710.05298

8 pages, 10 figures

arxiv created 2017/10/24 · arxiv updated 2017/10/25

Abstract

In this paper, we propose a generative model which learns the relationship between language and human action in order to generate a human action sequence given a sentence describing human behavior. The proposed generative model is a generative adversarial network (GAN), which is based on the sequence to sequence (SEQ2SEQ) model. Using the proposed generative network, we can synthesize various actions for a robot or a virtual agent using a text encoder recurrent neural network (RNN) and an action decoder RNN. The proposed generative network is trained from 29,770 pairs of actions and sentence annotations extracted from MSR-Video-to-Text (MSR-VTT), a large-scale video dataset. We demonstrate that the network can generate human-like actions which can be transferred to a Baxter robot, such that the robot performs an action based on a provided sentence. Results show that the proposed generative network correctly models the relationship between language and action and can generate a diverse set of actions from the same sentence.

Cited by

• MoSa: Motion Generation with Scalable Autoregressive Modeling
• MimicParts: Part-aware Style Injection for Speech-Driven 3D Motion Generation
• Text2Interact: High-Fidelity and Diverse Text-to-Two-Person Interaction Generation
• Embracing Aleatoric Uncertainty: Generating Diverse 3D Human Motion
• ViMoNet: A Multimodal Vision-Language Framework for Human Behavior Understanding from Motion and Video
• Generative AI-Driven High-Fidelity Human Motion Simulation
• Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data
• Motion Generation: A Survey of Generative Approaches and Benchmarks
• Motion-R1: Enhancing Motion Generation with Decomposed Chain-of-Thought and RL Binding
• DanceChat: Large Language Model-Guided Music-to-Dance Generation
• Hierarchical Intention-Aware Expressive Motion Generation for Humanoid Robots
• Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward
• MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation
• Shape My Moves: Text-Driven Shape-Aware Synthesis of Human Motions

Related

• Human Action Generation with Generative Adversarial Networks
• Actions Generation from Captions
• AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyond
• Summarizing a virtual robot's past actions in natural language
• Extracting Action Sequences from Texts Based on Deep Reinforcement Learning