AbsenceBench: Language Models Can't Tell What's Missing

2025/06/13 by Harvey Yiyun Fu, Aryan Shrivastava, Fu, Harvey Yiyun +9 · 21 voices · 4 citations
Computer Science · #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.11440

Abstract

Large language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a Haystack (NIAH) test. However, while models excel at recalling surprising information, they still struggle to identify clearly omitted information. We introduce AbsenceBench to assesses LLMs' capacity to detect missing information across three domains: numerical sequences, poetry, and GitHub pull requests. AbsenceBench asks models to identify which pieces of a document were deliberately removed, given access to both the original and edited contexts. Despite the apparent straightforwardness of these tasks, our experiments reveal that even state-of-the-art models like Claude-3.7-Sonnet achieve only 69.6% F1-score with a modest average context length of 5K tokens. Our analysis suggests this poor performance stems from a fundamental limitation: Transformer attention mechanisms cannot easily attend to "gaps" in documents since these absences don't correspond to any specific keys that can be attended to. Overall, our results and analysis provide a case study of the close proximity of tasks where models are already superhuman (NIAH) and tasks where models breakdown unexpectedly (AbsenceBench).

Citations

• FABLES: Evaluating faithfulness and content selection in book-length summarization
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Unlimiformer: Long-Range Transformers with Unlimited Length Input
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• The NarrativeQA Reading Comprehension Challenge
• Attention Is All You Need

Cited by

• Knowing What's Missing: Assessing Information Sufficiency in Question Answering
• Aligning Probabilistic Beliefs under Informative Missingness: LLM Steerability in Clinical Reasoning
• Mitigating hallucinations and omissions in LLMs for invertible problems: An application to hardware logic design automation
• Mapping Overlaps in Benchmarks through Perplexity in the Wild

Discussions

• AbsenceBench: Language models can't tell what's missing [hn, 324 points, 84 comments]
• LLMs excel at finding surprising “needles” in very long documents, but can they detect when information is conspicuously missing? 🫥AbsenceBench🫥 shows that even SoTA LLMs struggle on this task, sugg [bsky, 74 points, 2 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 (http://news.ycombinator.com/item?id=44332699) [bsky, 1 points, 0 comments]
• 🤔 Their results show that even advanced models struggle with finding missing information, especially when fewer things are missing. This means language models have a hard time noticing what is not th [bsky, 1 points, 0 comments]
• AbsenceBench: Language Models Can’t Tell What’s Missing arxiv.org/pdf/2506.11440 [bsky, 1 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing [bsky, 0 points, 0 comments]
• AbsenceBench: Language Models Can't Tell What's Missing #HackerNews https://arxiv.org/abs/2506.11440 [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 https://news.ycombinator.com/item?id=44332699 [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 arxiv.org [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 (https://news.ycombinator.com/item?id=44332699) [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: AbsenceBench: Language models can't tell what's missing [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 (http://news.ycombinator.com/item?id=44332699) [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2506.11440 この論文では、大規模言語モデルが欠落した情報を認識する能力について検証しています。 特に、常識的な知識や推論を必要とするタスクにおいて、その限界を明らかにしています。 新しいベンチマークデータセット「AbsenceBench」を用いて、モデルの性能を評価しています。 [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing view on hacker news [bsky, 0 points, 0 comments]
• AbsenceBench: Language Models Can't Tell What's Missing #llms #generativeai #benchmark #absence #absencebench #missing #information [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 (https://news.ycombinator.com/item?id=44332699) [bsky, 0 points, 0 comments]
• AbsenceBench: Language models can't tell what's missing https://arxiv.org/abs/2506.11440 (https://news.ycombinator.com/item?id=44332699) [bsky, 0 points, 0 comments]
• https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3ls3lxspt2ik2 [bsky, 0 points, 0 comments]
• "AbsenceBench: Language models can't tell what's missing" New language models struggle to find missing information in texts. Their understanding stops short when they can't see what's been left out, l [bsky, 0 points, 0 comments]

Related

• Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
• Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation
• Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?
• Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation
• Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration