Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

2017/03/30 by Rakshith Shetty, Marcus Rohrbach, Shetty, Rakshith +7 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1703.10476

openalex publication_date 2017/03/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While strong progress has been made in image captioning over the last years, machine and human captions are still quite distinct. A closer look reveals that this is due to the deficiencies in the generated word distribution, vocabulary size, and strong bias in the generators towards frequent captions. Furthermore, humans -- rightfully so -- generate multiple, diverse captions, due to the inherent ambiguity in the captioning task which is not considered in today's systems. To address these challenges, we change the training objective of the caption generator from reproducing groundtruth captions to generating a set of captions that is indistinguishable from human generated captions. Instead of handcrafting such a learning target, we employ adversarial training in combination with an approximate Gumbel sampler to implicitly match the generated distribution to the human one. While our method achieves comparable performance to the state-of-the-art in terms of the correctness of the captions, we generate a set of diverse captions, that are significantly less biased and match the word statistics better in several aspects.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Diversity-Promoting Objective Function for Neural Conversation Models
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
• Sequence Level Training with Recurrent Neural Networks
• Improved Techniques for Training GANs
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Categorical Reparameterization with Gumbel-Softmax
• Generating Visual Explanations
• Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models
• Deep Residual Learning for Image Recognition
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• Image Captioning with Semantic Attention

Cited by

• Surprisal reveals diversity gaps in image captioning and different scorers change the story
• Group-based Distinctive Image Captioning with Memory Attention
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Evaluating Robustness of Vision-Language Models Under Noisy Conditions

Related

• Generating Diverse and Accurate Visual Captions by Comparative Adversarial Learning
• Describing like humans: on diversity in image captioning
• Can adversarial training learn image captioning ?
• Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning
• Analysis of diversity-accuracy tradeoff in image captioning