Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

2025/07/10 by Sukjun Hwang, Brandon Wang, Hwang, Sukjun +3 · 14 voices · 14 citations
#cs.LG

paper · pdf · doi:10.48550/arxiv.2507.07955

Abstract

Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, to general models based on powerful architectures (e.g. the Transformer) that learn everything from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collection of new techniques that enable a dynamic chunking mechanism which automatically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchical network (H-Net) allows replacing the (implicitly hierarchical) tokenization-LM-detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, an H-Net with one stage of hierarchy operating at the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy to multiple stages further increases its performance by modeling multiple levels of abstraction, demonstrating significantly better scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robustness, and qualitatively learn meaningful data-dependent chunking strategies without any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end-to-end models that learn and scale better from unprocessed data.

Citations

• Log-Linear Attention
• Latent Iteration as Renormalization: Inference-Time Recurrence in a Depth-Recurrent Transformer Flows Attention Geometry Toward the SYK Conformal Fixed Point
• s1: Simple test-time scaling
• Byte Latent Transformer: Patches Scale Better Than Tokens
• Reasoning Beyond Words ? Exploring framework for hidden state reasoning
• An Empirical Study of Mamba-based Language Models
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
• Simple and Scalable Strategies to Continually Pre-train Large Language Models
• MambaByte: Token-free Selective State Space Model
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Training Compute-Optimal Large Language Models
• Hierarchical Transformers Are More Efficient Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Deep Residual Learning for Image Recognition
• ImageNet classification with deep convolutional neural networks

Cited by

• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space
• BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data
• Bolmo: Byteifying the Next Generation of Language Models
• MambaEye: A Size-Agnostic Visual Encoder with Causal Sequential Processing
• HieraMamba: Video Temporal Grounding via Hierarchical Anchor-Mamba Pooling
• Model-Aware Tokenizer Transfer
• Context-level Language Modeling by Learning Predictive Context Embeddings
• Xihe: Scalable Zero-Shot Time Series Learner Via Hierarchical Interleaved Block Attention
• Back to Bytes: Revisiting Tokenization Through UTF-8
• MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Models
• SuperThoughts: Reasoning Tokens in Superposition
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
• H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages
• ByteGen: A Tokenizer-Free Generative Model for Orderbook Events in Byte Space

Discussions

• Still not a lot of ML talk on bsky (at least in my feed), hence paper Sunday: my two most interesting recent reads - H Nets arxiv.org/abs/2507.07955 - Energy Based Transformers arxiv.org/abs/2507.0209 [bsky, 69 points, 5 comments]
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling [hn, 10 points, 0 comments]
• I try not to link to X, but I don't see this posted here. Pretty interesting arxiv.org/abs/2507.07955 x.com/sukjun_hwang... [bsky, 5 points, 0 comments]
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling [hn, 5 points, 0 comments]
• Ah ah. All here arxiv.org/pdf/2507.07955 But I recommend to read ByteLatentTransformer first arxiv.org/pdf/2412.09871 [bsky, 4 points, 1 comments]
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling [hn, 4 points, 0 comments]
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling [hn, 3 points, 0 comments]
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling [hn, 3 points, 0 comments]
• seems promising arxiv.org/pdf/2507.07955 [bsky, 2 points, 1 comments]
• multiscale & no frozen tokenizer! Dynamic chunking for end-to-end hierarchical sequence modeling (Hwang, Wang, and Gu) arxiv.org/abs/2507.07955 [bsky, 1 points, 1 comments]
• Language models can now learn to chunk text dynamically without any predefined tokenization rules. Read more: https://arxiv.org/abs/2507.07955 [bsky, 1 points, 0 comments]
• arxiv.org/abs/2507.07955 seems to be good, now whether it'll see uptake... [bsky, 1 points, 1 comments]
• I am confusion: arxiv.org/pdf/2507.07955 Like it's really interesting and great technique but the explanation seems to only apply to autoencoder type architecture. E.g. the decoding depends on input c [bsky, 0 points, 1 comments]
• arxiv.org/abs/2507.07955 [bsky, 0 points, 0 comments]

Related

• H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages
• Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
• Chunk-Distilled Language Modeling
• From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
• Structured Convergence in Large Language Model Representations via Hierarchical Latent Space Folding