Exploring Better Text Image Translation with Multimodal Codebook

2023/05/27 by Zhibin Lan, Lan, Zhibin, Jiawei Yu +13 · 6 citations
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Natural Language Processing Techniques #Translation Studies and Practices

paper · pdf · doi:10.48550/arxiv.2305.17415

openalex publication_date 2023/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Text image translation (TIT) aims to translate the source texts embedded in the image to target translations, which has a wide range of applications and thus has important research value. However, current studies on TIT are confronted with two main bottlenecks: 1) this task lacks a publicly available TIT dataset, 2) dominant models are constructed in a cascaded manner, which tends to suffer from the error propagation of optical character recognition (OCR). In this work, we first annotate a Chinese-English TIT dataset named OCRMT30K, providing convenience for subsequent studies. Then, we propose a TIT model with a multimodal codebook, which is able to associate the image with relevant texts, providing useful supplementary information for translation. Moreover, we present a multi-stage training framework involving text machine translation, image-text alignment, and TIT tasks, which fully exploits additional bilingual texts, OCR dataset and our OCRMT30K dataset to train our model. Extensive experiments and in-depth analyses strongly demonstrate the effectiveness of our proposed model and training framework.

Cited by

• Boosting Visual Knowledge-Intensive Training for LVLMs Through Causality-Driven Visual Object Completion
• LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation
• Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
• Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
• MT³: Scaling MLLM-based Text Image Machine Translation via Multi-Task Reinforcement Learning
• TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

Related

• Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task
• MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation
• PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Models
• Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
• E2TIMT: Efficient and Effective Modal Adapter for Text Image Machine Translation