Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

2018/09/14 by Deqing Sun, Xiaodong Yang, Sun, Deqing +5 · 3 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Advanced Vision and Imaging #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Processing Techniques and Applications

paper · pdf · doi:10.48550/arxiv.1809.05571

openalex publication_date 2018/09/14 · openalex created_date 2018/09/27 · openalex updated_date 2026/07/28

Abstract

We investigate two crucial and closely related aspects of CNNs for optical flow estimation: models and training. First, we design a compact but effective CNN model, called PWC-Net, according to simple and well-established principles: pyramidal processing, warping, and cost volume processing. PWC-Net is 17 times smaller in size, 2 times faster in inference, and 11% more accurate on Sintel final than the recent FlowNet2 model. It is the winning entry in the optical flow competition of the robust vision challenge. Next, we experimentally analyze the sources of our performance gains. In particular, we use the same training procedure of PWC-Net to retrain FlowNetC, a sub-network of FlowNet2. The retrained FlowNetC is 56% more accurate on Sintel final than the previously trained one and even 5% more accurate than the FlowNet2 model. We further improve the training procedure and increase the accuracy of PWC-Net on Sintel by 10% and on KITTI 2012 and 2015 by 20%. Our newly trained model parameters and training protocols will be available on https://github.com/NVlabs/PWC-Net

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradient-based learning applied to document recognition
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet classification with deep convolutional neural networks
• Multi-Scale Context Aggregation by Dilated Convolutions
• Deep Residual Learning for Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by

• A Study of Finetuning Video Transformers for Multi-view Geometry Tasks
• Removing Cost Volumes from Optical Flow Estimators
• FlowSeek: Optical Flow Made Easier with Depth Foundation Models and Motion Bases

Related

• FlowNet: Learning Optical Flow with Convolutional Networks
• PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
• A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization
• Disentangling Architecture and Training for Optical Flow
• Secrets in Computing Optical Flow by Convolutional Networks