Oddballness: universal anomaly detection with language models

2024/09/04 by Filip Graliński, Graliński, Filip, Ryszard Staruch +3 · 1 citation
Computer Science · #Advanced Malware Detection Techniques #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Spam and Phishing Detection

paper · pdf · doi:10.48550/arxiv.2409.03046

openalex publication_date 2024/09/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a new method to detect anomalies in texts (in general: in sequences of any data), using language models, in a totally unsupervised manner. The method considers probabilities (likelihoods) generated by a language model, but instead of focusing on low-likelihood tokens, it considers a new metric introduced in this paper: oddballness. Oddballness measures how ``strange'' a given token is according to the language model. We demonstrate in grammatical error detection tasks (a specific case of text anomaly detection) that oddballness is better than just considering low-likelihood events, if a totally unsupervised setup is assumed.

Cited by

• From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors

Related

• Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey
• Mechanistic Anomaly Detection for "Quirky" Language Models
• Anomaly Detection in Human Language via Meta-Learning: A Few-Shot Approach
• Variation and generality in encoding of syntactic anomaly information in sentence embeddings
• How is BERT surprised? Layerwise detection of linguistic anomalies