2023/09/07 by Ho Kei Cheng, Seoung Wug Oh, Cheng, Ho Kei +9 · 1 voice · 60 citations
Computer Science · #Artificial intelligence #Code (set theory) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Image segmentation #Multimodal Machine Learning Applications #Object (grammar) #Pattern recognition (psychology) #Scale-space segmentation #Segmentation #Segmentation-based object categorization #Task (project management) #Tracking (education) #Video tracking #Visual Attention and Saliency Detection #Vocabulary #cs.CV
paper · pdf · doi:10.48550/arxiv.2309.03903
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/09/07 · arxiv published 2023/09/07 · arxiv updated 2023/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Training data for video segmentation are expensive to annotate. This impedes extensions of end-to-end algorithms to new video segmentation tasks, especially in large-vocabulary settings. To 'track anything' without training on video data for every individual task, we develop a decoupled video segmentation approach (DEVA), composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. Due to this design, we only need an image-level model for the target task (which is cheaper to train) and a universal temporal propagation model which is trained once and generalizes across tasks. To effectively combine these two modules, we use bi-directional propagation for (semi-)online fusion of segmentation hypotheses from different frames to generate a coherent segmentation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks including large-vocabulary video panoptic segmentation, open-world video segmentation, referring video segmentation, and unsupervised video object segmentation. Code is available at: https://hkchengrex.github.io/Tracking-Anything-with-DEVA