Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance

2023/10/09 by Molly R. Petersen, Petersen, Molly R., van der Plas, Lonneke · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.05597

openalex publication_date 2023/10/09 · openalex created_date 2023/10/12 · openalex updated_date 2026/07/28

Abstract

While analogies are a common way to evaluate word embeddings in NLP, it is also of interest to investigate whether or not analogical reasoning is a task in itself that can be learned. In this paper, we test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans than those in commonly used NLP benchmarks. Our experiments find that models are able to learn analogical reasoning, even with a small amount of data. We additionally compare our models to a dataset with a human baseline, and find that after training, models approach human performance.

Cited by

• Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research

Related

• LLMs as Models for Analogical Reasoning
• Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
• Modelling Analogies and Analogical Reasoning: Connecting Cognitive Science Theory and NLP Research
• ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base
• Evaluating the Robustness of Analogical Reasoning in Large Language Models