Entropy-Driven Pre-Tokenization for Byte-Pair Encoding

2025/06/18 by Yifan Hu, Hu, Yifan, Frank Liang +11 · 1 voice
Computer Science · #Authorship Attribution and Profiling #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2506.15889

openalex publication_date 2025/06/18 · arxiv published 2025/06/18 · arxiv updated 2025/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such as Chinese presents significant challenges, as its frequency-driven merge operation is agnostic to linguistic boundaries. To address this, we propose two entropy-informed pre-tokenization strategies that guide BPE segmentation using unsupervised information-theoretic cues. The first approach uses pointwise mutual information and left/right entropy to identify coherent character spans, while the second leverages predictive entropy derived from a pretrained GPT-2 model to detect boundary uncertainty. We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE. Our results suggest that entropy-guided pre-tokenization not only enhances alignment with gold-standard linguistic units but also offers a promising direction for improving tokenization quality in low-resource and multilingual settings.

Citations

• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
• SuperBPE: Space Travel for Language Models
• Byte Latent Transformer: Patches Scale Better Than Tokens
• MorphPiece : A Linguistic Tokenizer for Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling
• Sub-Character Tokenization for Chinese Pretrained Language Models
• ByT5: Towards a token-free future with pre-trained byte-to-byte models
• UER: An Open-Source Toolkit for Pre-training Models
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Is Word Segmentation Necessary for Deep Learning of Chinese Representations?
• Neural Machine Translation of Rare Words with Subword Units

Discussions

• The second is on entropy-driven pre-tokenization for non-space delimited languages, arxiv.org/abs/2506.15889. That came out of a capstone project for a Harvard Masters program. Congrats to them on ach [bsky, 4 points, 0 comments]

Related

• From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• When Every Token Counts: Optimal Segmentation for Low-Resource Language Models
• Byte Pair Encoding is Suboptimal for Language Model Pretraining
• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier