TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

2025/12/23 by Gül Sena Altıntaş, Altıntaş, Gül Sena, Malikeh Ehghaghi +11
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · doi:10.48550/arxiv.2512.20757

openalex publication_date 2025/12/23 · openalex created_date 2025/12/26 · openalex updated_date 2026/07/28

Abstract

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we release fourteen pre-trained models that use different off-the-shelf tokenizers but are otherwise identical, using the same architecture, dataset, training budget, and initialization. We also release a multilingual robustness benchmark that measures model performance under real-world perturbations in English, Chinese, Farsi, Italian, and Turkish, curated by native annotators. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

Citations

• Bolmo: Byteifying the Next Generation of Language Models
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
• Qwen3 Technical Report
• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
• SuperBPE: Space Travel for Language Models
• Adversarial Tokenization
• How does a Language-Specific Tokenizer affect LLMs?
• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
• Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
• Byte Latent Transformer: Patches Scale Better Than Tokens
• Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
• GPT-4o System Card
• Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles
• Gemma 2: Improving Open Language Models at a Practical Size
• The Llama 3 Herd of Models
• Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Tokenization Falling Short: On Subword Robustness in Large Language Models
• Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
• Training LLMs over Neurally Compressed Text
• Token Alignment via Character Matching for Subword Completion
• Tokenization Is More Than Compression
• Tokenizer Choice For LLM Training: Negligible or Crucial?
• Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• GPT-4 Technical Report
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Few-shot Learning with Multilingual Language Models
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• ByT5: Towards a token-free future with pre-trained byte-to-byte models
• How Good is Your Tokenizer? On the Monolingual Performance of\n Multilingual Language Models
• mT5: A massively multilingual pre-trained text-to-text transformer
• PIQA: Reasoning about Physical Commonsense in Natural Language
• BPE-Dropout: Simple and Effective Subword Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• HellaSwag: Can a Machine Really Finish Your Sentence?
• XNLI: Evaluating Cross-lingual Sentence Representations
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Subword Regularization: Improving Neural Network Translation Models with\n Multiple Subword Candidates
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Machine Translation of Rare Words with Subword Units
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Beyond Text Compression: Evaluating Tokenizers Across Scales
• Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
• How does a Language-Specific Tokenizer affect LLMs?
• Bridging the Gap for Tokenizer-Free Language Models
• Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models