Estimating Tail Risks in Language Model Output Distributions

2026/04/24 by Rico Angell, Raghav Singhal, Zachary Horvitz +4
Computer Science · #Adversarial Robustness in Machine Learning #Focus (optics) #Importance sampling #Measure (data warehouse) #Natural Language Processing Techniques #Operationalization #Probabilistic logic #Probability distribution #Reliability (semiconductor) #Sampling (signal processing) #Statistical model #Topic Modeling #cs.AI #cs.LG

paper · pdf · open access · doi:10.48550/arxiv.2604.22167

published in arXiv (Cornell University) (Cornell University) · Accepted to ICML 2026

openalex publication_date 2026/04/24 · openalex created_date 2026/04/28 · openalex updated_date 2026/07/28 · arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

Citations

• Rare Event Analysis of Large Language Models
• Natural Emergent Misalignment from Reward Hacking in Production RL
• Estimating Worst-Case Frontier Risks of Open-Weight LLMs
• Sampling-aware Adversarial Attacks Against Large Language Models
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
• Forecasting Rare Language Model Behaviors
• Eliciting Language Model Behaviors with Investigator Agents
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
• Phi-4 Technical Report
• Steering Language Model Refusal with Sparse Autoencoders
• Estimating the Probabilities of Rare Outputs in Language Models
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
• AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs
• The Llama 3 Herd of Models
• External Invariants: A Cryptographic Trust Architecture for Institutional AI Inference
• Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
• A StrongREJECT for Empty Jailbreaks
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Steering Llama 2 via Contrastive Activation Addition
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Code of "Sirens' Whisper: Inaudible Near-Ultrasonic Jailbreaks of Speech-Driven LLMs"
• Probabilistic Querying of Continuous-Time Event Sequences
• Predictive Querying for Autoregressive Neural Sequence Models
• The sample size required in importance sampling