ByteSpan: Information-Driven Subword Tokenisation

2025/06/23 by Goriely, Zébulon, Salhan, Suchir, Lesci, Pietro +2
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.18639

Abstract

Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes - rather than pooling their representations - can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and Rényi efficiency for 25 languages.

Citations

• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Tokenization Falling Short: On Subword Robustness in Large Language Models
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Greed is All You Need: An Evaluation of Tokenizer Inference Methods
• Analyzing Cognitive Plausibility of Subword Tokenization
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Tokenization and the Noiseless Channel
• CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models
• MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
• The MiniPile Challenge for Data-Efficient Language Models
• Lost in Space Marking
• Improving Tokenisation by Alternative Treatment of Spaces
• Hierarchical Transformers Are More Efficient Language Models
• Scaling Vision Transformers
• How Good is Your Tokenizer? On the Monolingual Performance of\n Multilingual Language Models
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• BLiMP: The Benchmark of Linguistic Minimal Pairs for English
• Subword Regularization: Improving Neural Network Translation Models with\n Multiple Subword Candidates
• Attention Is All You Need
• Neural Machine Translation of Rare Words with Subword Units
• A Statistical Model for Word Discovery in Transcribed Speech
• An Efficient, Probabilistically Sound Algorithm for Segmentation and Word Discovery
• Statistical Learning by 8-Month-Old Infants
• Finding Structure in Time

Related

• Learn Your Tokens: Word-Pooled Tokenization for Language Modeling
• SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• Retrofitting Large Language Models with Dynamic Tokenization
• From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
• Multilingual Language Processing From Bytes