2020/07/12 by Aditya Mogadala, Marius Mosbach, Mogadala, Aditya +3
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #Video Analysis and Summarization #cs.CL #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2007.06077
International Conference on Machine Learning (ICML) 2020 Workshop (https://logicalreasoninggnn.github.io/)
arxiv created 2020/07/12 · openalex publication_date 2020/07/12 · arxiv updated 2020/07/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Generating longer textual sequences when conditioned on the visual information is an interesting problem to explore. The challenge here proliferate over the standard vision conditioned sentence-level generation (e.g., image or video captioning) as it requires to produce a brief and coherent story describing the visual content. In this paper, we mask this Vision-to-Sequence as Graph-to-Sequence learning problem and approach it with the Transformer architecture. To be specific, we introduce Sparse Graph-to-Sequence Transformer (SGST) for encoding the graph and decoding a sequence. The encoder aims to directly encode graph-level semantics, while the decoder is used to generate longer sequences. Experiments conducted with the benchmark image paragraph dataset show that our proposed achieve 13.3% improvement on the CIDEr evaluation measure when comparing to the previous state-of-the-art approach.