Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP

2024/08/08 by François Remy, Remy, François, Pieter Delobelle +9 · 10 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2408.04303

openalex publication_date 2024/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The development of monolingual language models for low and mid-resource languages continues to be hindered by the difficulty in sourcing high-quality training data. In this study, we present a novel cross-lingual vocabulary transfer strategy, trans-tokenization, designed to tackle this challenge and enable more efficient language adaptation. Our approach focuses on adapting a high-resource monolingual LLM to an unseen target language by initializing the token embeddings of the target language using a weighted average of semantically similar token embeddings from the source language. For this, we leverage a translation resource covering both the source and target languages. We validate our method with the Tweeties, a series of trans-tokenized LLMs, and demonstrate their competitive performance on various downstream tasks across a small but diverse set of languages. Additionally, we introduce Hydra LLMs, models with multiple swappable language modeling heads and embedding tables, which further extend the capabilities of our trans-tokenization strategy. By designing a Hydra LLM based on the multilingual model TowerInstruct, we developed a state-of-the-art machine translation model for Tatar, in a zero-shot manner, completely bypassing the need for high-quality parallel data. This breakthrough is particularly significant for low-resource languages like Tatar, where high-quality parallel data is hard to come by. By lowering the data and time requirements for training high-quality models, our trans-tokenization strategy allows for the development of LLMs for a wider range of languages, especially those with limited resources. We hope that our work will inspire further research and collaboration in the field of cross-lingual vocabulary transfer and contribute to the empowerment of languages on a global scale.

Cited by

• The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
• Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
• Model-Aware Tokenizer Transfer
• Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• Continually Adding New Languages to Multilingual Language Models
• A systematic evaluation of Dutch large language models’ surprisal estimates in sentence, paragraph and book reading
• LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations
• Conditional Unigram Tokenization with Parallel Data

Related

• Efficiently Adapting Pretrained Language Models To New Languages
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts
• Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer
• Exploring Design Choices for Building Language-Specific LLMs
• Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation