Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models

2024/05/08 by Sander Land, Max Bartolo, Land, Sander +1 · 3 voices · 7 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2405.05417

Abstract

The disconnect between tokenizer creation and model training in language models allows for specific inputs, such as the infamous SolidGoldMagikarp token, to induce unwanted model behaviour. Although such `glitch tokens', tokens present in the tokenizer vocabulary but that are nearly or entirely absent during model training, have been observed across various models, a reliable method to identify and address them has been missing. We present a comprehensive analysis of Large Language Model tokenizers, specifically targeting this issue of detecting under-trained tokens. Through a combination of tokenizer analysis, model weight-based indicators, and prompting techniques, we develop novel and effective methods for automatically detecting these problematic tokens. Our findings demonstrate the prevalence of such tokens across a diverse set of models and provide insights into improving the efficiency and safety of language models.

Cited by

• Shared Global and Local Geometry of Language Model Embeddings
• Bolmo: Byteifying the Next Generation of Language Models
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
• Vocabulary embeddings organize linguistic structure early in language model training
• A Cartography of Open Collaboration in Open Source AI: Mapping Practices, Motivations, and Governance in 14 Open Large Language Model Projects
• Alternatives To Next Token Prediction In Text Generation -- A Survey
• Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
• FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing

Discussions

• Automatically Detecting Under-Trained Tokens in Large Language Models [hn, 182 points, 26 comments]
• Automatically Detecting Under-Trained Tokens in Large Language Models (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• Tokenization for foundation models deserves its own workshop at an ML/NLP venue. So many issues can be traced back to tokenization. arxiv.org/abs/2405.05417 [bsky, 0 points, 0 comments]

Related

• Problematic Tokens: Tokenizer Bias in Large Language Models
• Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
• TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
• Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations