Less Is More: Reducing Token Counts Without Compromising Performance

2025/06/18 by Gyeongje Cho, Cho, Gyeongje, Y.T. So +6
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2506.15138

openalex publication_date 2025/06/18 · openalex created_date 2025/10/19 · openalex updated_date 2026/08/01

Abstract

Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost. Although longer, multi-word tokens can reduce fertility, naively adding them often degrades language model performance. We propose Thunder-Tok, a subword tokenizer that reduces fertility while preserving downstream performance. Thunder-Tok first constructs a large seed vocabulary from corpus substrings and filters structurally incomplete candidates, including invalid Unicode byte fragments and word-boundary violations. It then prunes the seed vocabulary using a likelihood-based token score derived from a uniform Jensen lower bound of the training-data probability. Experiments show that Thunder-Tok reduces fertility by approximately 25% in English and 9% in Korean compared with the standard BPE tokenizer while maintaining competitive performance.

Citations

• An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• GraphBPE: Molecular Graphs Meet Byte-Pair Encoding
• Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
• Getting the most out of your tokenizer for pre-training and domain adaptation
• Prompt Cache: Modular Attention Reuse for Low-Latency Inference
• Tokenizer Choice For LLM Training: Negligible or Crucial?
• Qwen Technical Report
• How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Language Model Tokenizers Introduce Unfairness Between Languages
• GPT-4 Technical Report
• KOBEST: Korean Balanced Evaluation of Significant Tasks
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• What Changes Can Large-scale Language Models Bring? Intensive Study on\n HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• How Good is Your Tokenizer? On the Monolingual Performance of\n Multilingual Language Models
• An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks
• BPE-Dropout: Simple and Effective Subword Regularization
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• HellaSwag: Can a Machine Really Finish Your Sentence?
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
• Subword Regularization: Improving Neural Network Translation Models with\n Multiple Subword Candidates
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Neural Machine Translation of Rare Words with Subword Units

Related

• How does a Language-Specific Tokenizer affect LLMs?
• The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
• Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models
• Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
• An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks