Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

2021/03/31 by Jiarui Xu, Xu, Jiarui, Xiaolong Wang +1 · 11 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #Video Surveillance and Tracking Methods #cs.CV

paper · pdf · doi:10.48550/arxiv.2103.17263

ICCV 2021 (oral). Project page and code: https://jerryxu.net/VFS

arxiv created 2021/10/14 · arxiv updated 2021/10/15

Abstract

Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired by the recent success in image-level contrastive learning and similarity learning for visual recognition. Our hypothesis is that if the representation is good for recognition, it requires the convolutional features to find correspondence between similar objects or parts. Our experiments show surprising results that VFS surpasses state-of-the-art self-supervised approaches for both OTB visual object tracking and DAVIS video object segmentation. We perform detailed analysis on what matters in VFS and reveals new properties on image and frame level similarity learning. Project page with code is available at https://jerryxu.net/VFS

Cited by

• Recurrent Video Masked Autoencoders
• A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking
• Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cues
• FlowFeat: Pixel-Dense Embedding of Motion Profiles
• Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics
• Representation Learning with Adaptive Superpixel Coding
• Predicting Video Slot Attention Queries from Random Slot-Feature Pairs
• Self-supervised pretraining of vision transformers for animal behavioral analysis and neural encoding
• Emergent Temporal Correspondences from Video Diffusion Transformers
• FRAME: Pre-Training Video Feature Representations via Anticipation and Memory
• Self-Supervised Spatial Correspondence Across Modalities

Related

• Contrastive Transformation for Self-supervised Correspondence Learning
• Learning Correspondence from the Cycle-Consistency of Time
• Self-supervised Learning for Video Correspondence Flow
• Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning
• Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning