InternVideo2: Scaling Foundation Models for Multimodal Video Understanding
2024/03/22 by Yi Wang, Kunchang Li, Wang, Yi +34 · 192 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2403.15377
openalex publication_date 2024/03/22 · openalex created_date 2024/03/26 · openalex updated_date 2026/07/28
Abstract
We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2/.
Cited by
- Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding
- Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representations
- M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
- Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
- IPCV: Information-Preserving Compression for MLLM Visual Encoders
- Atom: Efficient On-Device Video-Language Pipelines Through Modular Reuse
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs
- KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding
- Adapting MLLMs for Nuanced Video Retrieval
- DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
- HFS: Holistic Query-Aware Frame Selection for Efficient Video Reasoning
- AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Path
- UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models
- Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior
- PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglement
- DeRA: Decoupled Representation Alignment for Video Tokenization
- BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models
- TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
- WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning
- InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervision
- TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models
- HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics
- Seeing without Pixels: Perception from Camera Trajectories
- Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos
- LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models
- VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction
- Decoupling Perception from Reasoning for Hallucination-Resistant Video Understanding
- SciEducator: Scientific Video Understanding and Educating via Deming-Cycle Multi-Agent System
- MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection
- VITAL: Vision-Encoder-centered Pre-training for LMMs in Visual Quality Assessment
- Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination
- REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent
- VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning
- SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
- Learning Skill-Attributes for Transferable Assessment in Video
- Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models
- Do Blind Spots Matter for Word-Referent Mapping? A Computational Study with Infant Egocentric Video
- Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding
- TimeSense:Making Large Language Models Proficient in Time-Series Analysis
- Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition
- Cambrian-S: Towards Spatial Supersensing in Video
- Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers
- V-Agent: An Interactive Video Search System Using Vision-Language Models
- FOCUS: Efficient Keyframe Selection for Long Video Understanding
- Object-Aware 4D Human Motion Generation
- Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
- Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention
- World Simulation with Video Foundation Models for Physical AI
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
- A Video Is Not Worth a Thousand Words
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging
- Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges
- AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering
- From Preferences to Prejudice: The Role of Alignment Tuning in Shaping Social Bias in Video Diffusion Models
- Enrich and Detect: Video Temporal Grounding with Multimodal LLMs
- Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs
- K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding
- SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding
- Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting
- Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey
- Towards Safer and Understandable Driver Intention Prediction
- From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding
- AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understanding
- Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory
- Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
- Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
- Semantic Compression via Multimodal Representation Learning
- A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity
- VC-Agent: An Interactive Agent for Customized Video Dataset Collection
- MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoning
- VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception
- RefCaptioner: Multi-Reference Image-Grounded Video Captioning
- See What You Need: Query-Aware Visual Intelligence through Reasoning-Perception Loops
- TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
- SAIL-VL2 Technical Report
- ResidualViT for Efficient Temporally Dense Video Encoding
- Video Understanding by Design: How Datasets Shape Architectures and Insights
- AdsQA: Towards Advertisement Video Understanding
- Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening
- Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
- MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
- Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
- SUMMA: A Multimodal Large Language Model for Advertisement Summarization
- VimoRAG: Video-based Retrieval-augmented 3D Motion Generation for Motion Language Models
- KFFocus: Highlighting Keyframes for Enhanced Video Understanding
- MobileViCLIP: An Efficient Video-Text Model for Mobile Devices
- AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning
- A Survey on Video Temporal Grounding with Multimodal Large Language Model
- Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval
- Free-MoRef: Instantly Multiplexing Context Perception Capabilities of Video-MLLMs within Single Inference
- VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
- E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation
- Representation Shift: Unifying Token Compression with FlashAttention
- Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrieval
- Recognizing Actions from Robotic View for Natural Human-Robot Interaction
- Group Relative Augmentation for Data Efficient Action Detection
- HumanSAM: Classifying Human-centric Forgery Videos in Human Spatial, Appearance, and Motion Anomaly
- Object-centric Video Question Answering with Visual Grounding and Referring
- IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
- FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making
- Whom to Respond To? A Transformer-Based Model for Multi-Party Social Robot Interaction
- DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs
- Simplifying Traffic Anomaly Detection with Video Foundation Models
- Sparse-Dense Side-Tuner for efficient Video Temporal Grounding
- DisenQ: Disentangling Q-Former for Activity-Biometrics
- VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
- Detecting AI-Generated Videos with Spiking Neural Networks
- Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos
- Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
- DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
- EgoM2P: Egocentric Multimodal Multitask Pretraining
- Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
- LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
- Universal Retrieval for Multimodal Trajectory Modeling
- ImplicitQA: Going beyond frames towards Implicit Video Reasoning
- Task-Aware KV Compression For Cost-Effective Long Video Understanding
- Super Encoding Network: Recursive Association of Multi-Modal Encoders for Video Understanding
- DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
- Feature Hallucination for Self-supervised Action Recognition
- Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification
- SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis
- Beyond Audio and Pose: A General-Purpose Framework for Video Synchronization
- How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?
- video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
- Moment Sampling in Video LLMs for Long-Form Video QA
- EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization
- DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification
- DAVID-XR1: Detecting AI-Generated Videos with Explainable Reasoning
- DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs
- An Empirical study on LLM-based Log Retrieval for Software Engineering Metadata Management
- VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
- Q2E: Query-to-Event Decomposition for Zero-Shot Multilingual Text-to-Video Retrieval
- VideoMolmo: Spatio-Temporal Grounding Meets Pointing
- AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs
- DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
- MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs
- Artificial Behavior Intelligence: Technology, Challenges, and Future Directions
- DisTime: Distribution-based Time Representation for Video Large Language Models
- Time Blindness: Why Video-Language Models Can't See What Humans Can?
- Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review
- One Trajectory, One Token: Grounded Video Tokenization via Panoptic Sub-object Trajectory
- PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling
- Dynamic-Aware Video Distillation: Optimizing Temporal Resolution Based on Video Semantics
- A2Seek: Towards Reasoning-Centric Benchmark for Aerial Anomaly Understanding
- HuMoCon: Concept Discovery for Human Motion Understanding
- Think Before You Diffuse: Infusing Physical Rules into Video Diffusion
- HCQA-1.5 @ Ego4D EgoSchema Challenge 2025
- Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review
- Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
- RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models
- Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM
- Temporal Consistency Constrained Transferable Adversarial Attacks with Background Mixup for Action Recognition
- CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms
- Temporal Object Captioning for Street Scene Videos from LiDAR Tracks
- Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensembles
- Clapper: Compact Learning and Video Representation in VLMs
- Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency
- SafeVid: Toward Safety Aligned Video Large Multimodal Models
- Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs
- VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routing
- DiGIT: Multi-Dilated Gated Encoder and Central-Adjacent Region Integrated Decoder for Temporal Action Detection Transformer
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
- UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
- Learning Streaming Video Representation via Multitask Training
- Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction
- VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?
- Efficient Video Dataset Distillation via Cluster-Guided Prototype Blending
- ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding
- Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes
- Multimodal Lengthy Videos Retrieval Framework and Evaluation Metric
- VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT
- Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models
- Towards Understanding Camera Motions in Any Video
- LV-MAE: Learning Long Video Representations through Masked-Embedding Autoencoders
- VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment
- Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization
- KVAE: Family of Tokenizers for Multimodal Generative Models
- Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval
- Media Meets Communication in 6G: Fundamentals, Key Technologies, and Applications
- Multimodal Long Video Modeling Based on Temporal Dynamic Context
- VideoAds for Fast-Paced Video Understanding
- GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation
- VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning
- VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
- REVEAL: Relation-based Video Representation Learning for Video-Question-Answering
Related