GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models

2025/09/23 by Hutson, Dylan, Vennemeyer, Daniel, Deshmukh, Aneesh +2
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.19593

Abstract

We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43%. Prompting constraints guided by IG, such as enforcing question diversity, enable weaker models to significantly improve performance. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning.

Citations

• On Verbalized Confidence Scores for LLMs
• Do Large Language Models have Problem-Solving Capability under Incomplete Information Scenarios?
• Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain
• MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning
• Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling
• TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text
• Asking More Informative Questions for Grounded Retrieval
• Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
• NEWTON: Are Large Language Models Capable of Physical Reasoning?
• Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
• Large Language Models as Commonsense Knowledge for Large-Scale Task Planning
• Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals
• PROST: Physical Reasoning of Objects through Space and Time
• Language Models are Few-Shot Learners
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Acquiring Common Sense Spatial Knowledge through Implicit Spatial Templates
• Verb Physics: Relative Physical Knowledge of Actions and Objects
• Twenty (simple) questions
• Improving Hypernymy Detection with an Integrated Path-based and Distributional Method
• Basic objects in natural categories

Related

• Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain
• Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling
• Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models
• Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
• The Consensus Game: Language Model Generation via Equilibrium Search