Summarizing Videos with Attention

2018/12/05 by Fajtl, Jiri, Sokeh, Hajar Sadeghi, Argyriou, Vasileios +2 · 3 citations
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.1812.01969

Abstract

In this work we propose a novel method for supervised, keyshots based video summarization by applying a conceptually simple and computationally efficient soft, self-attention mechanism. Current state of the art methods leverage bi-directional recurrent networks such as BiLSTM combined with attention. These networks are complex to implement and computationally demanding compared to fully connected networks. To that end we propose a simple, self-attention based network for video summarization which performs the entire sequence to sequence transformation in a single feed forward pass and single backward pass during training. Our method sets a new state of the art results on two benchmarks TvSum and SumMe, commonly used in this domain.

Cited by

• MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing
• Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation
• SummDiff: Generative Modeling of Video Summarization with Diffusion

Related

• Video Summarization with Attention-Based Encoder-Decoder Networks
• Self-Attention Based Generative Adversarial Networks For Unsupervised Video Summarization
• TRIM: A Self-Supervised Video Summarization Framework Maximizing Temporal Relative Information and Representativeness
• Video Summarization Using Fully Convolutional Sequence Networks
• Progressive Video Summarization via Multimodal Self-supervised Learning