Deep Reinforcement Learning-based Image Captioning with Embedding Reward

2017/04/12 by Zhou Ren, Ren, Zhou, Xiaoyu Wang +7 · 8 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1704.03899

openalex publication_date 2017/04/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Image captioning is a challenging problem owing to the complexity in understanding the image content and diverse ways of describing it in natural language. Recent advances in deep neural networks have substantially improved the performance of this task. Most state-of-the-art approaches follow an encoder-decoder framework, which generates captions using a sequential recurrent prediction model. However, in this paper, we introduce a novel decision-making framework for image captioning. We utilize a "policy network" and a "value network" to collaboratively generate captions. The policy network serves as a local guidance by providing the confidence of predicting the next word according to the current state. Additionally, the value network serves as a global and lookahead guidance by evaluating all possible extensions of the current state. In essence, it adjusts the goal of predicting the correct words towards the goal of generating captions similar to the ground truth captions. We train both networks using an actor-critic reinforcement learning model, with a novel reward defined by visual-semantic embedding. Extensive experiments and analyses on the Microsoft COCO dataset show that the proposed framework outperforms state-of-the-art approaches across different evaluation metrics.

Citations

• Adam: A Method for Stochastic Optimization
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Long Short-Term Memory
• Sequence Level Training with Recurrent Neural Networks
• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
• Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models
• Deep Residual Learning for Image Recognition
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Deep Compositional Captioning: Describing Novel Object Categories\n without Paired Training Data
• CIDEr: Consensus-based Image Description Evaluation
• Image Captioning with Semantic Attention

Cited by

• PRPO: Paragraph-level Policy Optimization for Vision-Language Deepfake Detection
• FFNet: Video Fast-Forwarding via Reinforcement Learning
• Hierarchical Photo-Scene Encoder for Album Storytelling
• A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping
• Reconstruction Network for Video Captioning
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Deep Reinforcement Learning with Label Embedding Reward for Supervised Image Hashing
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning

Related

• Image Captioning based on Deep Reinforcement Learning
• Actor-Critic Sequence Training for Image Captioning
• A sequential guiding network with attention for image captioning
• Image Captioning Based on a Hierarchical Attention Mechanism and Policy Gradient Optimization
• Multi-modal reward for visual relationships-based image captioning