Real-time Animation Generation and Control on Rigged Models via Large Language Models

2023/10/27 by Han Huang, Fernanda De La Torre, Huang, Han +9 · 3 citations
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Graphics (cs.GR) #Handwritten Text Recognition Techniques #Human Motion and Animation #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2310.17838

openalex publication_date 2023/10/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a novel method for real-time animation control and generation on rigged models using natural language input. First, we embed a large language model (LLM) in Unity to output structured texts that can be parsed into diverse and realistic animations. Second, we illustrate LLM's potential to enable flexible state transition between existing animations. We showcase the robustness of our approach through qualitative results on various rigged models and motions.

Cited by

• Exploring User Acceptance and Concerns toward LLM-powered Conversational Agents in Immersive Extended Reality
• I Can't Join, But I Will Send My Agent: Stand-in Enhanced Asynchronous Meetings (SEAM)
• SonoCraftAR: Towards Supporting Personalized Authoring of Sound-Reactive AR Interfaces by Deaf and Hard of Hearing Users

Related

• MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
• MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding
• Programmable Motion Generation for Open-Set Motion Control Tasks
• Story-to-Motion: Synthesizing Infinite and Controllable Character Animation from Long Text
• FreeMotion: MoCap-Free Human Motion Synthesis with Multimodal Large Language Models