Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

2018/04/25 by Gunnar A. Sigurdsson, Sigurdsson, Gunnar A., Abhinav Gupta +7 · 14 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Video Analysis and Summarization #Video Surveillance and Tracking Methods #cs.CV

paper · pdf · doi:10.48550/arxiv.1804.09626

openalex publication_date 2018/04/25 · arxiv created 2018/04/30 · arxiv updated 2018/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Charades-Ego has temporal annotations and textual descriptions, making it suitable for egocentric video classification, localization, captioning, and new tasks utilizing the cross-modal nature of the data.

Citations

• First-Person Activity Forecasting with Online Inverse Reinforcement Learning
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

Cited by

• Supervised Contrastive Frame Aggregation for Video Representation Learning
• EventBench: Towards Comprehensive Benchmarking of Event-based MLLMs
• TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding
• Hi-OSCAR: Hierarchical Open-set Classifier for Human Activity Recognition
• Video Understanding by Design: How Datasets Shape Architectures and Insights
• Diffusion-Based Action Recognition Generalizes to Untrained Domains
• OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planning
• EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception
• EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization
• EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
• MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs
• Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs
• EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
• Towards Generalizing Temporal Action Segmentation to Unseen Views

Related

• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos
• Retrieval-Augmented Egocentric Video Captioning
• Comparing Learning Paradigms for Egocentric Video Summarization
• Making Third Person Techniques Recognize First-Person Actions in Egocentric Videos
• Action Scene Graphs for Long-Form Understanding of Egocentric Videos