Deep learning for video classification and captioning

2016/09/30 by Zuxuan Wu, Ting Yao, Yanwei Fu +1
Computer Science · #Advanced Image and Video Retrieval Techniques #CLIPS #Closed captioning #Comprehension #Deep learning #Focus (optics) #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Natural language #Semantics (computer science) #Video retrieval #cs.CV #cs.MM

paper · pdf · doi:10.1145/3122865.3122867

Book chapter in Frontiers of Multimedia Research

openalex created_date 2016/10/07 · openalex publication_date 2017/12/19 · arxiv created 2018/02/22 · arxiv updated 2018/02/23 · openalex updated_date 2026/08/05

Abstract

Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data. In this paper, we focus on reviewing two lines of research aiming to stimulate the comprehension of videos with deep learning: video classification and video captioning. While video classification concentrates on automatically labeling video clips based on their semantic contents like human actions or complex events, video captioning attempts to generate a complete and natural sentence, enriching the single label as in video classification, to capture the most informative dynamics in videos. In addition, we also provide a review of popular benchmarks and competitions, which are critical for evaluating the technical progress of this vibrant field.

Citations

• An argument for basic emotions
• A logical calculus of the ideas immanent in nervous activity
• Situation Awareness Is Adaptive, Externally Directed Consciousness
• A set of level 3 basic linear algebra subprograms
• Algorithm 778: L-BFGS-B
• Situation Awareness: A Critical But Ill-Defined Phenomenon
• An Algorithm for Finding Best Matches in Logarithmic Expected Time
• A saliency-based search mechanism for overt and covert shifts of visual attention
• Aesthetics and Emotions in Images
• Long Short-Term Memory
• Statistical Models for Earthquake Occurrences and Residual Analysis for Point Processes
• Spectra of some self-exciting and mutually exciting point processes
• Turing computability with neural nets
• An Information-Maximization Approach to Blind Separation and Blind Deconvolution
• Situation awareness global assessment technique (SAGAT)
• Solving the multiple instance problem with axis-parallel rectangles
• LabelMe: A Database and Web-Based Tool for Image Annotation
• ImageNet Large Scale Visual Recognition Challenge
• Content-based image retrieval at the end of the early years
• A Fast Learning Algorithm for Deep Belief Nets
• Distinctive Image Features from Scale-Invariant Keypoints
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Vision-guided robot hearing
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• MapReduce
• YFCC100M
• Robust Real-Time Face Detection
• Probabilistic encryption
• A method for obtaining digital signatures and public-key cryptosystems
• Quantization