Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

2023/10/05 by Remy, François, Delobelle, Pieter, Berendt, Bettina +2 · 5 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2310.03477

Abstract

Training monolingual language models for low and mid-resource languages is made challenging by limited and often inadequate pretraining data. In this study, we propose a novel model conversion strategy to address this issue, adapting high-resources monolingual language models to a new target language. By generalizing over a word translation dictionary encompassing both the source and target languages, we map tokens from the target tokenizer to semantically similar tokens from the source language tokenizer. This one-to-many token mapping improves tremendously the initialization of the embedding table for the target language. We conduct experiments to convert high-resource models to mid- and low-resource languages, namely Dutch and Frisian. These converted models achieve a new state-of-the-art performance on these languages across all sorts of downstream tasks. By reducing significantly the amount of data and time required for training state-of-the-art models, our novel model conversion strategy has the potential to benefit many languages worldwide.

Cited by

• The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems
• Model-Aware Tokenizer Transfer
• MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch
• Conditional Unigram Tokenization with Parallel Data
• Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

Related

• Efficiently Adapting Pretrained Language Models To New Languages
• Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• Exploring Design Choices for Building Language-Specific LLMs
• One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers