TVQA: Localized, Compositional Video Question Answering
2018/09/05 by Jie Lei, Lei, Jie, Licheng Yu +5 · 44 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.1809.01696
openalex publication_date 2018/09/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.
Citations
Cited by
- Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization
- LongVideoAgent: Multi-Agent Reasoning with Long Videos
- AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding
- Evaluating the Capability of Video Question Generation for Expert Knowledge Elicitation
- HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering
- VisualActBench: Can VLMs See and Act like a Human?
- IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval
- CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
- REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
- CourseTimeQA: A Lecture-Video Benchmark and a Latency-Constrained Cross-Modal Fusion Method for Timestamped QA
- VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection
- Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search
- Multi-speaker Attention Alignment for Multimodal Social Interaction
- CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models
- VideoChain: A Transformer-Based Framework for Multi-hop Video Question Generation
- SRNN: Spatiotemporal Relational Neural Network for Intuitive Physics Understanding
- LiveStar: Live Streaming Assistant for Real-World Online Video Understanding
- NVIDIA Nemotron Nano V2 VL
- Semantic Frame Aggregation-based Transformer for Live Video Comment Generation
- StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
- Multimedia-Aware Question Answering: A Review of Retrieval and Cross-Modal Reasoning Architectures
- CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
- InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
- Not in Sync: Unveiling Temporal Bias in Audio Chat Models
- SocialNLI: A Dialogue-Centric Social Inference Dataset
- POVQA: Preference-Optimized Video Question Answering with Rationales for Data Efficiency
- What You See is What You Ask: Evaluating Audio Descriptions
- FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learning
- Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm
- VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
- Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
- Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media
- A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
- Cinéaste: A Fine-grained Contextual Movie Question Answering Benchmark
- MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
- Video Understanding by Design: How Datasets Shape Architectures and Insights
- AdsQA: Towards Advertisement Video Understanding
- Video-LLMs with Temporal Visual Screening
- MovieCORE: COgnitive REasoning in Movies
- Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
- JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics
- FineBadminton: A Multi-Level Dataset for Fine-Grained Badminton Video Understanding
- Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation
Related