Learning Video Representations using Contrastive Bidirectional Transformer

2019/06/13 by Chen Sun, Sun, Chen, Fabien Baradel +5 · 2 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.1906.05743

openalex publication_date 2019/06/13 · openalex created_date 2019/10/03 · openalex updated_date 2026/07/28

Abstract

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.

Citations

• UCF-101: A dataset of 101 human actions classes from videos in the wild
• Long Short-Term Memory
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Exploring the Limits of Language Modeling
• The Kinetics Human Action Video Dataset
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Learning deep representations by mutual information estimation and maximization
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Contrastive Multiview Coding
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million\n Narrated Video Clips

Cited by

• Support-set bottlenecks for video-text representation learning
• About contrastive unsupervised representation learning for classification and its convergence

Related

• VideoBERT: A Joint Model for Video and Language Representation Learning
• Cross-Architecture Self-supervised Video Representation Learning
• Composable Augmentation Encoding for Video Representation Learning
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• Self-supervised Co-training for Video Representation Learning