2017/11/30 by Tianfan Xue, Baian Chen, Jiajun Wu +2 · 4 citations
Computer Science · #cs.CV
paper · pdf · doi:10.1007/s11263-018-01144-2
published as International Journal of Computer Vision (IJCV), 127(8):1106-1125, 2019 · IJCV 2019. Project page: http://toflow.csail.mit.edu
arxiv created 2019/11/10 · arxiv updated 2019/11/12
Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.