Multimodal Machine Translation with Visual Scene Graph Pruning

2025/05/26 by Chenyu Lu, Shiliang Sun, Lu, Chenyu +9 · 1 citation
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.19507

openalex publication_date 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multimodal machine translation (MMT) seeks to address the challenges posed by linguistic polysemy and ambiguity in translation tasks by incorporating visual information. A key bottleneck in current MMT research is the effective utilization of visual data. Previous approaches have focused on extracting global or region-level image features and using attention or gating mechanisms for multimodal information fusion. However, these methods have not adequately tackled the issue of visual information redundancy in MMT, nor have they proposed effective solutions. In this paper, we introduce a novel approach--multimodal machine translation with visual Scene Graph Pruning (PSG), which leverages language scene graph information to guide the pruning of redundant nodes in visual scene graphs, thereby reducing noise in downstream translation tasks. Through extensive comparative experiments with state-of-the-art methods and ablation studies, we demonstrate the effectiveness of the PSG model. Our results also highlight the promising potential of visual information pruning in advancing the field of MMT.

Citations

• Adding Multimodal Capabilities to a Text-only Translation Model
• Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
• CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation
• Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination
• ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation
• Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation
• VALHALLA: Visual Hallucination for Machine Translation
• Neural Machine Translation with Phrase-Level Universal Visual Representations
• On Vision Features in Multimodal Machine Translation
• Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Learning Transferable Visual Models From Natural Language Supervision
• COMET: A Neural Framework for MT Evaluation
• A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
• Unbiased Scene Graph Generation from Biased Training
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Auto-Encoding Scene Graphs for Image Captioning
• Scene Graph Parsing as Dependency Parsing
• Attention Is All You Need
• Imagination improves Multimodal Translation
• Doubly-Attentive Decoder for Multi-modal Neural Machine Translation
• Incorporating Global Visual Features into Attention-Based Neural Machine Translation
• CUNI System for WMT16 Automatic Post-Editing and Multimodal Translation Tasks
• Multi30K: Multilingual English-German Image Descriptions
• Neural Machine Translation of Rare Words with Subword Units
• Adam: A Method for Stochastic Optimization

Cited by

• ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning

Related

• GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation
• Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets
• Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination
• Probing the Need for Visual Context in Multimodal Machine Translation
• Multimodal Machine Translation through Visuals and Speech