Byte Latent Transformer: Patches Scale Better Than Tokens

2024/12/13 by Artidoro Pagnoni, Ram Pasunuru, Pagnoni, Artidoro +27 · 17 voices · 39 citations
Computer Science · #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2412.09871

openalex publication_date 2024/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM\narchitecture that, for the first time, matches tokenization-based LLM\nperformance at scale with significant improvements in inference efficiency and\nrobustness. BLT encodes bytes into dynamically sized patches, which serve as\nthe primary units of computation. Patches are segmented based on the entropy of\nthe next byte, allocating more compute and model capacity where increased data\ncomplexity demands it. We present the first FLOP controlled scaling study of\nbyte-level models up to 8B parameters and 4T training bytes. Our results\ndemonstrate the feasibility of scaling models trained on raw bytes without a\nfixed vocabulary. Both training and inference efficiency improve due to\ndynamically selecting long patches when data is predictable, along with\nqualitative improvements on reasoning and long tail generalization. Overall,\nfor fixed inference costs, BLT shows significantly better scaling than\ntokenization-based models, by simultaneously growing both patch and model size.\n

Cited by

• Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
• The Art of Not Forgetting
• Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content
• Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya
• Continuous Autoregressive Language Models
• Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
• GQ-VAE: A gated quantized VAE for learning variable length tokens
• TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
• Bolmo: Byteifying the Next Generation of Language Models
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• Length-MAX Tokenizer for Language Models
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging
• Human-aligned Quantification of Numerical Data
• Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits
• Attention and Compression is all you need for Controllably Efficient Language Models
• The Art of Not Forgetting A Local Learning Architecture for Continual Learning
• TweetyBERT: Automated parsing of birdsong through self-supervised machine learning
• Model-Aware Tokenizer Transfer
• Context-level Language Modeling by Learning Predictive Context Embeddings
• Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
• See the Text: From Tokenization to Visual Reading
• Back to Bytes: Revisiting Tokenization Through UTF-8
• Don't Throw Away Your Pretrained Model
• Latent Speech-Text Transformer
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• LongTail-Swap: benchmarking language models' abilities on rare words
• EntroPE: Entropy-Guided Dynamic Patch Encoder for Time Series Forecasting
• Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
• SuperThoughts: Reasoning Tokens in Superposition
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
• <i>De novo</i> Design of All-atom Biomolecular Interactions with RFdiffusion3
• Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music
• Revisiting associative recall in modern recurrent models
• SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
• DeCAL Tokenwise Compression
• MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Data
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• ByteGen: A Tokenizer-Free Generative Model for Orderbook Events in Byte Space

Discussions

• Byte latent transformer: Patches scale better than tokens (2024) [hn, 107 points, 22 comments]
• What is Meta's BLT? Byte Latent Transformer(BLT) BLT encodes bytes into dynamic patches using light-weight local models and processes them with a large latent transformer. Think of it as a transform [bsky, 15 points, 0 comments]
• here’s the original paper from Meta that was all research, this is a legit impl with competitive performance that you can download and try today arxiv.org/abs/2412.09871 [bsky, 5 points, 0 comments]
• arxiv.org/abs/2412.09871 this one is the more recent one i am thinking of, there's another one called byt5 that's older and not hierarchical [bsky, 4 points, 1 comments]
• *Byte Latent Transformer: Patches Scale Better Than Tokens* by @artidoro.bsky.social et al. Trains a small encoder to dynamically aggregate bytes into tokens, which are input to a standard autoregres [bsky, 4 points, 0 comments]
• This gives me the same energy as one of my favourite AI papers: arxiv.org/abs/2412.09871 (It doesn’t seem like it ever went anywhere though?) [bsky, 4 points, 1 comments]
• Byte Latent Transformer: Patches Scale Better Than Tokens https://arxiv.org/abs/2412.09871 [bsky, 0 points, 0 comments]
• Interesting approach to sequence modeling! Exploring alternative data representations could unlock gains in efficiency and performance for long-form data. 🤖 #ai Byte Latent Transformer: Patches Scale [bsky, 0 points, 0 comments]
• https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3loyqhq7oq5m2 [bsky, 0 points, 0 comments]
• Am I on the wrong thought path in thinking that Byte Latent Transformers are what these realtime multimodal models that power Gemini 2 are based on? arxiv.org/abs/2412.09871 [bsky, 0 points, 0 comments]
• ... Patches - also in der Unterscheidung von Token -, die LLM besser skalieren als per Tokenization. Das wird mit einer verschachtelten LLM-Architektur erreicht: ein ganz normaler Transformer als Kern [bsky, 0 points, 1 comments]
• Byte Latent Transformer: Patches Scale Better Than Tokens https://arxiv.org/abs/2412.09871 (https://news.ycombinator.com/item?id=43965099) [bsky, 0 points, 0 comments]
• Byte Latent Transformer: Patches Scale Better Than Tokens https://arxiv.org/abs/2412.09871 (https://news.ycombinator.com/item?id=43965099) [bsky, 0 points, 0 comments]
• FAIR team demonstrates PoC for Byte Latent Transformer (BLT), a robust, efficient LLM that processes raw bytes, enhancing multilingual capabilities by scaling both model parameter and patch size. #AI [bsky, 0 points, 0 comments]
• Byte Latent Transformer: Patches Scale Better Than Tokens [bsky, 0 points, 0 comments]
• Byte Latent Transformer: Patches Scale Better Than Tokens #HackerNews https://arxiv.org/abs/2412.09871 [bsky, 0 points, 0 comments]
• Byte latent transformer: Patches scale better than tokens (2024) https://arxiv.org/abs/2412.09871 https://news.ycombinator.com/item?id=43965099 [bsky, 0 points, 0 comments]

Related

• Beyond Next Token Prediction: Patch-Level Training for Large Language Models
• Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
• Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles
• Getting the most out of your tokenizer for pre-training and domain adaptation
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity