2020/09/16 by Erika Lu, Forrester Cole, Lu, Erika +13 · 4 citations
Computer Science · #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Graphics (cs.GR) #Human Pose and Action Recognition #Video Analysis and Summarization #cs.CV #cs.GR
paper · pdf · doi:10.48550/arxiv.2009.07833
In SIGGRAPH Asia 2020. Project webpage: https://retiming.github.io/. Added references
openalex publication_date 2020/09/16 · arxiv created 2021/10/01 · arxiv updated 2021/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present a method for retiming people in an ordinary, natural video -- manipulating and editing the time in which different motions of individuals in the video occur. We can temporally align different motions, change the speed of certain actions (speeding up/slowing down, or entirely "freezing" people), or "erase" selected people from the video altogether. We achieve these effects computationally via a dedicated learning-based layered video representation, where each frame in the video is decomposed into separate RGBA layers, representing the appearance of different people in the video. A key property of our model is that it not only disentangles the direct motions of each person in the input video, but also correlates each person automatically with the scene changes they generate -- e.g., shadows, reflections, and motion of loose clothing. The layers can be individually retimed and recombined into a new video, allowing us to achieve realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions and involving multiple individuals, including dancing, trampoline jumping, or group running.