2022/01/27 by Saikat Dutta, Dutta, Saikat, Arulkumar Subramaniam +3 · 1 citation
Computer Science · #Advanced Image Processing Techniques #Advanced Vision and Imaging #Artificial Intelligence (cs.AI) #Artificial intelligence #Block-matching algorithm #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #FOS: Computer and information sciences #Image (mathematics) #Image warping #Interpolation (computer graphics) #Linear interpolation #Linear motion #Motion (physics) #Motion compensation #Motion estimation #Motion field #Motion interpolation #Optical flow #Pattern recognition (psychology) #Pixel #Quarter-pixel motion #Video Coding and Compression Technologies #Video processing #Video tracking #cs.AI #cs.CV
paper · pdf · doi:10.48550/arxiv.2201.11407
published in arXiv (Cornell University) (Cornell University) · Accepted at CLIC workshop, CVPR 2022. Code: https://github.com/saikatdutta/NME-VFI
openalex publication_date 2022/01/27 · arxiv created 2022/04/13 · arxiv updated 2022/04/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Video frame interpolation aims to synthesize one or multiple frames between two consecutive frames in a video. It has a wide range of applications including slow-motion video generation, frame-rate up-scaling and developing video codecs. Some older works tackled this problem by assuming per-pixel linear motion between video frames. However, objects often follow a non-linear motion pattern in the real domain and some recent methods attempt to model per-pixel motion by non-linear models (e.g., quadratic). A quadratic model can also be inaccurate, especially in the case of motion discontinuities over time (i.e. sudden jerks) and occlusions, where some of the flow information may be invalid or inaccurate. In our paper, we propose to approximate the per-pixel motion using a space-time convolution network that is able to adaptively select the motion model to be used. Specifically, we are able to softly switch between a linear and a quadratic model. Towards this end, we use an end-to-end 3D CNN encoder-decoder architecture over bidirectional optical flows and occlusion maps to estimate the non-linear motion model of each pixel. Further, a motion refinement module is employed to refine the non-linear motion and the interpolated frames are estimated by a simple warping of the neighboring frames with the estimated per-pixel motion. Through a set of comprehensive experiments, we validate the effectiveness of our model and show that our method outperforms state-of-the-art algorithms on four datasets (Vimeo, DAVIS, HD and GoPro).