Generating Descriptions for Sequential Images with Local-Object Attention and Global Semantic Context Modelling

2020/12/02 by Su, Jing, Lin, Chenghua, Zhou, Mian +2
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2012.01295

Abstract

In this paper, we propose an end-to-end CNN-LSTM model for generating descriptions for sequential images with a local-object attention mechanism. To generate coherent descriptions, we capture global semantic context using a multi-layer perceptron, which learns the dependencies between sequential images. A paralleled LSTM network is exploited for decoding the sequence descriptions. Experimental results show that our model outperforms the baseline across three different evaluation metrics on the datasets published by Microsoft.

Related

• A sequential guiding network with attention for image captioning
• You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet
• Generating Image Sequence from Description with LSTM Conditional GAN
• Recurrent Soft Attention Model for Common Object Recognition
• Semantic Object Parsing with Local-Global Long Short-Term Memory