Self-Supervised Video Representation Learning With Odd-One-Out Networks

2016/11/21 by Basura Fernando, Hakan Bilen, Fernando, Basura +6
Computer Science · Mathematics · #Action (physics) #Annotation #Anomaly Detection Techniques and Applications #Artificial intelligence #Artificial neural network #Bounded function #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Convolutional neural network #FOS: Computer and information sciences #Feature learning #Human Pose and Action Recognition #Machine learning #Mathematics #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Representation (politics) #Set (abstract data type) #Subsequence #Supervised learning #Task (project management) #cs.CV

paper · pdf · doi:10.48550/arxiv.1611.06646

Accepted in In IEEE International Conference on Computer Vision and Pattern Recognition CVPR 2017

openalex publication_date 2016/11/21 · arxiv created 2017/04/05 · arxiv updated 2017/04/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

We propose a new self-supervised CNN pre-training technique based on a novel auxiliary task called "odd-one-out learning". In this task, the machine is asked to identify the unrelated or odd element from a set of otherwise related elements. We apply this technique to self-supervised video representation learning where we sample subsequences from videos and ask the network to learn to predict the odd video subsequence. The odd video subsequence is sampled such that it has wrong temporal order of frames while the even ones have the correct temporal order. Therefore, to generate a odd-one-out question no manual annotation is required. Our learning machine is implemented as multi-stream convolutional neural network, which is learned end-to-end. Using odd-one-out networks, we learn temporal representations for videos that generalizes to other related tasks such as action recognition. On action classification, our method obtains 60.3% on the UCF101 dataset using only UCF101 data for training which is approximately 10% better than current state-of-the-art self-supervised learning methods. Similarly, on HMDB51 dataset we outperform self-supervised state-of-the art methods by 12.7% on action classification task.

Citations

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Auto-association by multilayer perceptrons and singular value decomposition
• Gradient-based learning applied to document recognition
• ImageNet Large Scale Visual Recognition Challenge
• Slow Feature Analysis: Unsupervised Learning of Invariances
• Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
• Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimising Global Loss Functions
• VideoLSTM Convolves, Attends and Flows for Action Recognition
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Action Recognition with Dynamic Image Networks
• Learning to See by Moving
• Learning Spatiotemporal Features with 3D Convolutional Networks
• MatConvNet - Convolutional Neural Networks for MATLAB
• Unsupervised Learning of Visual Representations using Videos

Related

• Self-Supervised Video Representation Learning by Video Incoherence Detection
• Exploring Relations in Untrimmed Videos for Self-Supervised Learning
• Shuffle and Learn: Unsupervised Learning using Temporal Order Verification
• A Large-Scale Analysis on Self-Supervised Video Representation Learning
• Video Representation Learning by Recognizing Temporal Transformations