InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
2023/07/13 by Yi Wang, Wang, Yi, Yinan He +26 · 145 citations
Computer Science · #Multimodal Machine Learning Applications #Human Pose and Action Recognition #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2307.06942
Abstract
This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.
Cited by
- TimePLE: Rethinking Temporal Representation for Video Temporal Grounding
- MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning
- Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
- Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations
- StoryMem: Multi-shot Long Video Storytelling with Memory
- The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text
- EasyV2V: A High-quality Instruction-based Video Editing Framework
- VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
- IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning
- Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure
- Adapting MLLMs for Nuanced Video Retrieval
- SneakPeek: Future-Guided Instructional Streaming Video Generation
- JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing
- AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Path
- BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models
- FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
- TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection
- ShotDirector: Directorially Controllable Multi-Shot Video Generation with Cinematographic Transitions
- VABench: A Comprehensive Benchmark for Audio-Video Generation
- Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learning
- OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory
- TV2TV: A Unified Framework for Interleaved Language and Video Generation
- Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression
- GeoVideo: Introducing Geometric Regularization into Video Generation Model
- PAI-Bench: A Comprehensive Benchmark For Physical AI
- InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
- AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation
- TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models
- MVAD : A Comprehensive Multimodal Video-Audio Dataset for AIGC Detection
- Video-CoM: Interactive Video Reasoning via Chain of Manipulations
- Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding
- Layer-Aware Video Composition via Split-then-Merge
- ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Access
- Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
- TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
- Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation
- Calibrated Multimodal Representation Learning with Missing Modalities
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition
- Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum
- LoCoT2V-Bench: A Benchmark for Long-Form and Complex Text-to-Video Generation
- FreeSliders: Training-Free, Modality-Agnostic Concept Sliders for Fine-Grained Diffusion Control in Images, Audio, and Video
- CineWeaver: Training-Free Reference-Controllable Multi-Shot Long Video Generation for Cinematic Storytelling
- Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
- Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models
- Listening without Looking: Modality Bias in Audio-Visual Captioning
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
- HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives
- Video Consistency Distance: Enhancing Temporal Consistency for Image-to-Video Generation via Reward-Based Fine-Tuning
- Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model
- From Preferences to Prejudice: The Role of Alignment Tuning in Shaping Social Bias in Video Diffusion Models
- InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
- VidGuard-R1: AI-Generated Video Detection and Explanation via Reasoning MLLMs and RL
- InternSVG: Towards Unified SVG Tasks with Multimodal Large Language Models
- ESCA: Contextualizing Embodied Agents via Scene-Graph Generation
- Ctrl-VI: Controllable Video Synthesis via Variational Inference
- From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding
- Character Mixing for Video Generation
- When and Where do Events Switch in Multi-Event Video Generation?
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
- NeMo: Needle in a Montage for Video-Language Understanding
- Embodied Representation Alignment with Mirror Neurons
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- 4DHumanDiff: Direct Text-to-4DGS Generation for Consistent 360-Degree Dynamic Humans
- ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment
- TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
- A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
- Dense Video Understanding with Gated Residual Tokenization
- Enhancing Video Large Language Models with Structured Multi-Video Collaborative Reasoning
- Effectively obtaining acoustic, visual and textual data from videos
- MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
- CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models
- Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation
- Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation
- Yan: Foundational Interactive Video Generation
- KFFocus: Highlighting Keyframes for Enhanced Video Understanding
- TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding
- MobileViCLIP: An Efficient Video-Text Model for Mobile Devices
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity
- A Survey on Video Temporal Grounding with Multimodal Large Language Model
- Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigm
- GAIS: Frame-Level Gated Audio-Visual Integration with Semantic Variance-Scaled Perturbation for Text-Video Retrieval
- AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation
- Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval
- MagicAnime: A Hierarchically Annotated, Multimodal and Multitasking Dataset with Benchmarks for Cartoon Animation Generation
- Datasets and Recipes for Video Temporal Grounding via Reinforcement Learning
- VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
- Principled Multimodal Representation Learning
- Vidar: Embodied Video Diffusion Model for Generalist Manipulation
- Product of Experts for Visual Generation
- Semantic Frame Interpolation
- LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data
- Detecting AI-Generated Videos with Spiking Neural Networks
- UVLM: Benchmarking Video Language Model for Underwater World Understanding
- CI-VID: A Coherent Interleaved Text-Video Dataset
- Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations
- TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency
- FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusion
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant
- Beyond Audio and Pose: A General-Purpose Framework for Video Synchronization
- Audio-Sync Video Generation with Multi-Stream Temporal Control
- EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization
- Fake it till You Make it: Reward Modeling as Discriminative Prediction
- UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions
- Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models
- Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision
- Towards Holistic Visual Quality Assessment of AI-Generated Videos: A LLM-Based Multi-Dimensional Evaluation Model
- ContentV: Efficient Training of Video Generation Models with Limited Compute
- AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation
- MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs
- VUDG: A Dataset for Video Understanding Domain Generalization
- DisTime: Distribution-based Time Representation for Video Large Language Models
- What Do Latent Action Models Actually Learn?
- OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation
- Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
- InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO
- CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms
- Temporal Object Captioning for Street Scene Videos from LiDAR Tracks
- A Challenge to Build Neuro-Symbolic Video Agents
- Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding
- LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding
- Video-GPT via Next Clip Diffusion
- SafeVid: Toward Safety Aligned Video Large Multimodal Models
- LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation
- Generative Pre-trained Autoregressive Diffusion Transformer
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
- SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models
- Helios: Real Real-Time Long Video Generation Model
- Learning Streaming Video Representation via Multitask Training
- Action100M: A Large-scale Video Action Dataset
- T2VAttack: Adversarial Attack on Text-to-Video Diffusion Models
- BadVideo: Stealthy Backdoor Attack against Text-to-Video Generation
- VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
- Video-Bench: Human-Aligned Video Generation Benchmark
- Multimodal Lengthy Videos Retrieval Framework and Evaluation Metric
- LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- ReSpec: Relevance and Specificity Grounded Online Filtering for Learning on Video-Text Data Streams
- Grounding-MD: Grounded Video-language Pre-training for Open-World Moment Detection
- Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models
- Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval
- Mavors: Multi-granularity Video Representation for Multimodal Large Language Model
- VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding
- SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning
Related