Ad-hoc Concept Forming in the Game Codenames as a Means for Evaluating Large Language Models

2025/02/17 by Sherzod Hakimov, Hakimov, Sherzod, Lara Pfennigschmidt +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.11707

openalex publication_date 2025/02/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This study utilizes the game Codenames as a benchmarking tool to evaluate large language models (LLMs) with respect to specific linguistic and cognitive skills. LLMs play each side of the game, where one side generates a clue word covering several target words and the other guesses those target words. We designed various experiments by controlling the choice of words (abstract vs. concrete words, ambiguous vs. monosemic) or the opponent (programmed to be faster or slower in revealing words). Recent commercial and open-weight models were compared side-by-side to find out factors affecting their performance. The evaluation reveals details about their strategies, challenging cases, and limitations of LLMs.

Related

• Codenames as a Benchmark for Large Language Models
• Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game
• Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboard
• Do You Get the Hint? Benchmarking LLMs on the Board Game Concept
• Large Language Models and Games: A Survey and Roadmap