End-to-End Learning Using Cycle Consistency for Image-to-Caption Transformations

2019/03/25 by Keisuke Hagiwara, Hagiwara, Keisuke, Yusuke Mukuta +3
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.1903.10118

arxiv created 2019/03/25 · arxiv updated 2019/03/26

Abstract

So far, research to generate captions from images has been carried out from the viewpoint that a caption holds sufficient information for an image. If it is possible to generate an image that is close to the input image from a generated caption, i.e., if it is possible to generate a natural language caption containing sufficient information to reproduce the image, then the caption is considered to be faithful to the image. To make such regeneration possible, learning using the cycle-consistency loss is effective. In this study, we propose a method of generating captions by learning end-to-end mutual transformations between images and texts. To evaluate our method, we perform comparative experiments with and without the cycle consistency. The results are evaluated by an automatic evaluation and crowdsourcing, demonstrating that our proposed method is effective.

Related

• C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
• Improving Generalization of Image Captioning with Unsupervised Prompt Learning
• Cycle-Consistency Learning for Captioning and Grounding
• Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences
• Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning