Response: Emergent analogical reasoning in large language models

2023/08/30 by Damian Hodel, Jevin West, Hodel, Damian +2 · 1 voice · 9 citations
Computer Science · Mathematics · Psychology · Social Sciences · #Analogical reasoning #Analogy #Artificial intelligence #Cognitive psychology #Cognitive science #Computer science #Counterexample #Epistemology #Field (mathematics) #Language and cultural evolution #Linguistics #Mathematics #Memorization #Natural Language Processing Techniques #Philosophy #Physics #Psychology #Range (aeronautics) #Shot (pellet) #Simple (philosophy) #String (physics) #Theoretical physics #Topic Modeling #Zero (linguistics) #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2308.16118

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/08/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In their recent Nature Human Behaviour paper, "Emergent analogical reasoning in large language models," (Webb, Holyoak, and Lu, 2023) the authors argue that "large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems." In this response, we provide counterexamples of the letter string analogies. In our tests, GPT-3 fails to solve simplest variations of the original tasks, whereas human performance remains consistently high across all modified versions. Zero-shot reasoning is an extraordinary claim that requires extraordinary evidence. We do not see that evidence in our experiments. To strengthen claims of humanlike reasoning such as zero-shot reasoning, it is important that the field develop approaches that rule out data memorization.

Cited by

• Do AI Models Perform Human-like Abstract Reasoning Across Modalities?
• Genomic Next-Token Predictors are In-Context Learners
• Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research

Discussions

• Have large language models developed analogical reasoning skills? Maybe, but our simple experiments demonstrate otherwise: arxiv.org/abs/2308.16118 [bsky, 51 points, 0 comments]

Related

• Emergent Analogical Reasoning in Large Language Models
• Evidence from counterfactual tasks supports emergent analogical reasoning in large language models
• Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
• Evaluating the Robustness of Analogical Reasoning in Large Language Models
• Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction