Imagination improves Multimodal Translation

2017/05/11 by Desmond Elliott, Elliott, Desmond, Ákos Kádár +1 · 7 citations
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.1705.04350

Clarified main contributions, minor correction to Equation 8, additional comparisons in Table 2, added more related work

openalex publication_date 2017/05/11 · arxiv created 2017/07/07 · arxiv updated 2017/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.

Citations

• Bidirectional recurrent neural networks
• Multilingual Image Description with Neural Sequence Models
• Keystroke dynamics as signal for shallow syntactic parsing
• Neural Machine Translation with Latent Semantic of Image and Text
• Rethinking the Inception Architecture for Computer Vision
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Learning language through pictures

Cited by

• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
• FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks
• Dual-branch Prompting for Multimodal Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Emergent Translation in Multi-Agent Communication
• Multimodal Machine Translation with Visual Scene Graph Pruning
• TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

Related

• Generative Imagination Elevates Machine Translation
• Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation
• A Visual Attention Grounding Neural Model for Multimodal Machine Translation
• Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation
• Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translation