SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation

2021/01/21 by Duke, Brendan, Ahmed, Abdalla, Wolf, Christian +2 · 4 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2101.08833

Abstract

In this paper we introduce a Transformer-based approach to video object segmentation (VOS). To address compounding error and scalability issues of prior work, we propose a scalable, end-to-end method for VOS called Sparse Spatiotemporal Transformers (SST). SST extracts per-pixel representations for each object in a video using sparse attention over spatiotemporal features. Our attention-based formulation for VOS allows a model to learn to attend over a history of multiple frames and provides suitable inductive bias for performing correspondence-like computations necessary for solving motion segmentation. We demonstrate the effectiveness of attention-based over recurrent networks in the spatiotemporal domain. Our method achieves competitive results on YouTube-VOS and DAVIS 2017 with improved scalability and robustness to occlusions compared with the state of the art. Code is available at https://github.com/dukebw/SSTVOS.

Cited by

• Segment Anything Across Shots: A Method and Benchmark
• MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes
• Advancing Complex Video Object Segmentation via Progressive Concept Construction
• A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

Related

• Hierarchical Spatiotemporal Transformers for Video Object Segmentation
• TransVOS: Video Object Segmentation with Transformers
• Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation
• Spatiotemporal CNN for Video Object Segmentation
• Associating Objects with Transformers for Video Object Segmentation