2023/03/10 by Xingyi Li, Zhiguo Cao, Li, Xingyi +9 · 2 citations
Computer Science · #Advanced Vision and Imaging #Animation #Art #Artificial Intelligence (cs.AI) #Artificial intelligence #Computer Graphics and Visualization Techniques #Computer Vision and Pattern Recognition (cs.CV) #Computer animation #Computer graphics (images) #Computer science #Computer vision #FOS: Computer and information sciences #Feature (linguistics) #Graphics (cs.GR) #Image (mathematics) #Optical flow #Photography #Point cloud #Rendering (computer graphics) #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2303.05724
openalex publication_date 2023/03/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present 3D Cinemagraphy, a new technique that marries 2D image animation with 3D photography. Given a single still image as input, our goal is to generate a video that contains both visual content animation and camera motion. We empirically find that naively combining existing 2D image animation and 3D photography methods leads to obvious artifacts or inconsistent animation. Our key insight is that representing and animating the scene in 3D space offers a natural solution to this task. To this end, we first convert the input image into feature-based layered depth images using predicted depth values, followed by unprojecting them to a feature point cloud. To animate the scene, we perform motion estimation and lift the 2D motion into the 3D scene flow. Finally, to resolve the problem of hole emergence as points move forward, we propose to bidirectionally displace the point cloud as per the scene flow and synthesize novel views by separately projecting them into target image planes and blending the results. Extensive experiments demonstrate the effectiveness of our method. A user study is also conducted to validate the compelling rendering results of our method.