Pun Unintended: LLMs and the Illusion of Humor Understanding

2025/09/15 by Alessandro Zangari, Zangari, Alessandro, Matteo Marcuzzo +8 · 1 voice · 1 citation
Computer Science · Psychology · #68T50 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Humor Studies and Applications #I.2.7 #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2509.12158

openalex publication_date 2025/09/15 · arxiv published 2025/09/15 · arxiv updated 2025/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Puns are a form of humorous wordplay that exploits polysemy and phonetic similarity. While LLMs have shown promise in detecting puns, we show in this paper that their understanding often remains shallow, lacking the nuanced grasp typical of human interpretation. By systematically analyzing and reformulating existing pun benchmarks, we demonstrate how subtle changes in puns are sufficient to mislead LLMs. Our contributions include comprehensive and nuanced pun detection benchmarks, human evaluation across recent LLMs, and an analysis of the robustness challenges these models face in processing puns.

Citations

• Qwen2.5 Technical Report
• The Llama 3 Herd of Models
• A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians
• "A good pun is its own reword": Can Large Language Models Understand Puns?
• How Ambiguous are the Rationales for Natural Language Reasoning? A Simple Approach to Handling Rationale Uncertainty
• AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms
• We're Afraid Language Models Aren't Modeling Ambiguity
• Can Pre-trained Language Models Understand Chinese Humor?
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• ExPUNations: Augmenting Puns with Keywords and Explanations
• A Unified Framework for Pun Generation with Humor Principles
• Large Language Models are Zero-Shot Reasoners
• AmbiPun: Generating Humorous Puns with Ambiguous Context
• ePiC: Employing Proverbs in Context as a Benchmark for Abstract Language Understanding
• Measuring Association Between Labels and Free-Text Rationales
• Language Models are Few-Shot Learners
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Efficient Estimation of Word Representations in Vector Space

Cited by

• Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

Discussions

• Paper link arxiv.org/abs/2509.12158 [bsky, 3 points, 0 comments]

Related

• "A good pun is its own reword": Can Large Language Models Understand Puns?
• Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes
• "The Boating Store Had Its Best Sail Ever": Pronunciation-attentive Contextualized Pun Recognition
• ExPUNations: Augmenting Puns with Keywords and Explanations
• A Survey of Pun Generation: Datasets, Evaluations and Methodologies