2022/10/15 by Hengsheng Zhang, Zhang, Hengsheng, Xueyi Zou +10 · 1 citation
Computer Science · Engineering · #Advanced Image Processing Techniques #Advanced Vision and Imaging #Algorithm #Artificial intelligence #Codec #Computation #Computer Vision and Pattern Recognition (cs.CV) #Computer hardware #Computer science #Computer vision #Encoding (memory) #FOS: Computer and information sciences #Feature (linguistics) #Frame (networking) #Image (mathematics) #Image Processing Techniques and Applications #Motion compensation #Motion estimation #Motion vector #Optical flow #Pixel #Quarter-pixel motion #Video processing #cs.CV
paper · pdf · doi:10.48550/arxiv.2210.08229
published in arXiv (Cornell University) (Cornell University)
arxiv created 2022/10/15 · openalex publication_date 2022/10/15 · arxiv updated 2022/10/18 · openalex created_date 2022/10/20 · openalex updated_date 2026/08/08
Online processing of compressed videos to increase their resolutions attracts increasing and broad attention. Video Super-Resolution (VSR) using recurrent neural network architecture is a promising solution due to its efficient modeling of long-range temporal dependencies. However, state-of-the-art recurrent VSR models still require significant computation to obtain a good performance, mainly because of the complicated motion estimation for frame/feature alignment and the redundant processing of consecutive video frames. In this paper, considering the characteristics of compressed videos, we propose a Codec Information Assisted Framework (CIAF) to boost and accelerate recurrent VSR models for compressed videos. Firstly, the framework reuses the coded video information of Motion Vectors to model the temporal relationships between adjacent frames. Experiments demonstrate that the models with Motion Vector based alignment can significantly boost the performance with negligible additional computation, even comparable to those using more complex optical flow based alignment. Secondly, by further making use of the coded video information of Residuals, the framework can be informed to skip the computation on redundant pixels. Experiments demonstrate that the proposed framework can save up to 70% of the computation without performance drop on the REDS4 test videos encoded by H.264 when CRF is 23.