Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer

2023/05/23 by Elizabeth Salesky, Neha Verma, Salesky, Elizabeth +5 · 3 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.14280

openalex publication_date 2023/05/23 · openalex created_date 2023/05/27 · openalex updated_date 2026/07/28

Abstract

We introduce and demonstrate how to effectively train multilingual machine translation models with pixel representations. We experiment with two different data settings with a variety of language and script coverage, demonstrating improved performance compared to subword embeddings. We explore various properties of pixel representations such as parameter sharing within and across scripts to better understand where they lead to positive transfer. We observe that these properties not only enable seamless cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. We hope this work contributes to more extensible multilingual models for all languages and scripts.

Cited by

• CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare
• FLEXITOKENS: Flexible Tokenization for Evolving Language Models
• Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations

Related

• Multilingual Pretraining for Pixel Language Models
• Language Modelling with Pixels
• Overcoming Vocabulary Constraints with Pixel-level Fallback
• Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models
• Cross-lingual Word Embeddings beyond Zero-shot Machine Translation