DrawTalking: Building Interactive Worlds by Sketching and Speaking

2024/01/11 by Rosenberg, Karl Toby, Kazi, Rubaiat Habib, Wei, Li-Yi +2 · 6 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #D.1.7 #D.2.2 #Emerging Technologies (cs.ET) #FOS: Computer and information sciences #Graphics (cs.GR) #H.5.1 #H.5.2 #Human-Computer Interaction (cs.HC) #I.2.7

paper · doi:10.48550/arxiv.2401.05631

Abstract

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Cited by

• TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech
• Sketch Animation: State-of-the-art Report
• Vistoria: A Multimodal System to Support Fictional Story Writing through Instrumental Text-Image Co-Editing
• Interaction as Intelligence: Deep Research With Human-AI Partnership
• ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent
• MapStory: Prototyping Editable Map Animations with LLM Agents

Related

• TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech
• SketchAgent: Language-Driven Sequential Sketch Generation
• Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications
• Learning to Draw: Emergent Communication through Sketching
• WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI