2018/11/26 by Marcella Cornia, Cornia, Marcella, Lorenzo Baraldi +3 · 8 citations
Computer Science · #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.1811.10652
Current captioning approaches can describe images using black-box\narchitectures whose behavior is hardly controllable and explainable from the\nexterior. As an image can be described in infinite ways depending on the goal\nand the context at hand, a higher degree of controllability is needed to apply\ncaptioning algorithms in complex scenarios. In this paper, we introduce a novel\nframework for image captioning which can generate diverse descriptions by\nallowing both grounding and controllability. Given a control signal in the form\nof a sequence or set of image regions, we generate the corresponding caption\nthrough a recurrent architecture which predicts textual chunks explicitly\ngrounded on regions, following the constraints of the given control.\nExperiments are conducted on Flickr30k Entities and on COCO Entities, an\nextended version of COCO in which we add grounding annotations collected in a\nsemi-automatic manner. Results demonstrate that our method achieves state of\nthe art performances on controllable image captioning, in terms of caption\nquality and diversity. Code and annotations are publicly available at:\nhttps://github.com/aimagelab/show-control-and-tell.\n