Spelling Bee Embeddings for Language Modeling

2026/01/25 by Markus N. Rabe, Judith Clymo, Zheren Dong · 1 voice
#cs.LG

paper · pdf

Abstract

We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss.

Citations

• Large Language Models Lack Understanding of Character Composition of Words
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Training Compute-Optimal Large Language Models
• Memorizing Transformers
• Attention Is All You Need

Discussions

• shit someone finally added character information directly into the llm. i was looking at finally rigging this up but they already did it arxiv.org/abs/2601.180... [bsky, 44 points, 5 comments]

Related

• Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
• Improving Text Embeddings with Large Language Models
• Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters
• Word Embeddings Are Steers for Language Models
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe