Imagination improves Multimodal Translation

2017/05/11 by Desmond Elliott, Elliott, Desmond, Ákos Kádár +1 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1705.04350

openalex publication_date 2017/05/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.

Citations

• Bidirectional recurrent neural networks
• Multilingual Image Description with Neural Sequence Models
• Keystroke dynamics as signal for shallow syntactic parsing
• Neural Machine Translation with Latent Semantic of Image and Text
• Rethinking the Inception Architecture for Computer Vision
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Learning language through pictures

Cited by

• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
• FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks

Related

• Generative Imagination Elevates Machine Translation
• Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation
• A Visual Attention Grounding Neural Model for Multimodal Machine Translation
• Exploiting Multimodal Reinforcement Learning for Simultaneous Machine\n Translation
• Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translation