VideoBERT: A Joint Model for Video and Language Representation Learning

2019/04/03 by Chen Sun, Austin Myers, Sun, Chen +7 · 52 citations
Computer Science · #Multimodal Machine Learning Applications #Human Pose and Action Recognition #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.1904.01766

Abstract

Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

Cited by

• Towards High-Level Semantic Intelligence
• FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing
• Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
• Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation
• EEA: Exploration-Exploitation Agent for Long Video Understanding
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• A Visuospatial Dataset for Naturalistic Verb Learning
• Temporal Query Networks for Fine-grained Video Understanding
• Unifying Vision-and-Language Tasks via Text Generation
• Multi-step Predictive Coding Leads To Simplicity Bias
• Spatio-Temporal Data Enhanced Vision-Language Model for Traffic Scene Understanding
• Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection
• MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer
• FOCUS: Efficient Keyframe Selection for Long Video Understanding
• Masked Diffusion Captioning for Visual Feature Learning
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• MERLOT: Multimodal Neural Script Knowledge Models
• Video Understanding as Machine Translation
• Accurate Word Representations with Universal Visual Guidance
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection
• Modest-Align: Data-Efficient Alignment for Vision-Language Models
• Multimedia-Aware Question Answering: A Review of Retrieval and Cross-Modal Reasoning Architectures
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
• Prompt-Guided Spatial Understanding with RGB-D Transformers for Fine-Grained Object Relation Reasoning
• AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Expressive and Scalable Quantum Fusion for Multimodal Learning
• Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• ProfVLM: A Lightweight Video-Language Model for Multi-View Proficiency Estimation
• Answer-checking in Context: A Multi-modal FullyAttention Network for Visual Question Answering
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• RefCaptioner: Multi-Reference Image-Grounded Video Captioning
• Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
• iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning
• Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media
• MVQA-68K: A Multi-dimensional and Causally-annotated Dataset with Quality Interpretability for Video Assessment
• Video Understanding by Design: How Datasets Shape Architectures and Insights
• Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes
• TrajSV: A Trajectory-based Model for Sports Video Representations and Applications
• SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition
• Adversarial Video Promotion Against Text-to-Video Retrieval
• IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning

Related

• ActBERT: Learning Global-Local Video-Text Representations
• FILS: Self-Supervised Video Feature Prediction In Semantic Language Space
• Learning Video Representations from Large Language Models
• Distilling Vision-Language Models on Millions of Videos
• Learning Video Representations using Contrastive Bidirectional Transformer