2024/10/21 by Gengshan Yang, Yang, Gengshan, Andrea Bajcsy +5 · 1 citation
Mathematics · Physics and Astronomy · Psychology · #COVID-19 epidemiological studies #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Graphics (cs.GR) #Mental Health Research Topics #Opinion Dynamics and Social Influence #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2410.16259
openalex publication_date 2024/10/21 · openalex created_date 2024/11/06 · openalex updated_date 2026/07/28
We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone.