2021/09/26 by Haozhi Cao, Cao, Haozhi, Yuecong Xu +11 · 1 citation
Computer Science · Mathematics · #Artificial intelligence #CLIPS #Coherence (philosophical gambling strategy) #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #FOS: Computer and information sciences #Feature learning #Human Pose and Action Recognition #Machine learning #Mathematics #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Representation (politics) #Video Analysis and Summarization #cs.CV
paper · pdf · doi:10.48550/arxiv.2109.12493
published in arXiv (Cornell University) (Cornell University) · 11 pages, 7 figures
arxiv created 2021/09/26 · openalex publication_date 2021/09/26 · arxiv updated 2021/09/28 · openalex created_date 2021/10/11 · openalex updated_date 2026/08/05
This paper introduces a novel self-supervised method that leverages incoherence detection for video representation learning. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between incoherent clips from the same raw video. We evaluate our proposed method through extensive experiments on action recognition and video retrieval utilizing various backbone networks. Experiments show that our proposed method achieves state-of-the-art performance across different backbone networks and different datasets compared with previous coherence-based methods.