Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

2015/10/26 by Haonan Yu, Jiang Wang, Yu, Haonan +7 · 2 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition #Multimodal Machine Learning Applications #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.1510.07712

openalex publication_date 2015/10/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present an approach that exploits hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem, i.e., generating one or multiple sentences to describe a realistic video. Our hierarchical framework contains a sentence generator and a paragraph generator. The sentence generator produces one simple short sentence that describes a specific short video interval. It exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. The paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator. We evaluate our approach on two large-scale benchmark datasets: YouTubeClips and TACoS-MultiLevel. The experiments demonstrate that our approach significantly outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and 0.305 respectively.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• A Neural Conversational Model
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Multi-scale Multiple Instance Video Description Network
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Long Short-Term Memory
• Finding Structure in Time
• ImageNet Large Scale Visual Recognition Challenge
• Learning a Recurrent Visual Representation for Image Caption Generation
• Sequence to Sequence Learning with Neural Networks
• Bidirectional recurrent neural networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence Level Training with Recurrent Neural Networks
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Question Answering with Subgraph Embeddings
• Jointly Modeling Embedding and Translation to Bridge Video and Language
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Cited by

• Dense Video Captioning using Graph-based Sentence Summarization
• Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization

Related

• Video captioning with recurrent networks based on frame- and video-level features and visual content classification
• MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
• Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
• Sequence to Sequence -- Video to Text
• GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning