Mechanistic Anomaly Detection for "Quirky" Language Models

2025/04/09 by David Johnston, Johnston, David O., Nora Belrose +2 · 4 citations
Computer Science · #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Network Security and Intrusion Detection #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2504.08812

openalex publication_date 2025/04/09 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

As LLMs grow in capability, the task of supervising LLMs becomes more challenging. Supervision failures can occur if LLMs are sensitive to factors that supervisors are unaware of. We investigate Mechanistic Anomaly Detection (MAD) as a technique to augment supervision of capable models; we use internal model features to identify anomalous training signals so they can be investigated or discarded. We train detectors to flag points from the test environment that differ substantially from the training environment, and experiment with a large variety of detector features and scoring rules to detect anomalies in a set of ``quirky'' language models. We find that detectors can achieve high discrimination on some tasks, but no detector is effective across all models and tasks. MAD techniques may be effective in low-stakes applications, but advances in both detection and evaluation are likely needed if they are to be used in high stakes settings.

Citations

• Sycophancy in Large Language Models: Causes and Mitigations
• Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
• Scaling and evaluating sparse autoencoders
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• Eliciting Latent Knowledge from Quirky Language Models
• Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations
• Benchmarks for Detecting Measurement Tampering
• Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
• ASSET: Robust Backdoor Data Detection Across a Multiplicity of Deep Learning Paradigms
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• SPECTRE: Defending Against Backdoor Attacks Using Robust Statistics
• Deep Probabilistic Models to Detect Data Poisoning Attacks
• STRIP: A Defence Against Trojan Attacks on Deep Neural Networks
• Variational Inference with Normalizing Flows
• UFID: A Unified Framework for Input-level Backdoor Detection on Diffusion Models

Cited by

• Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies

Related

• Anomaly Detection of Tabular Data Using LLMs
• Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey
• Large Language Models for Anomaly Detection in Computational Workflows: from Supervised Fine-Tuning to In-Context Learning
• Oddballness: universal anomaly detection with language models
• LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesis