How to evaluate the cognitive abilities of LLMs

2025/01/15 by Anna A. Ivanova · 1 voice · 15 citations
Computer Science · Psychology · Social Sciences · #Artificial Intelligence in Law #Cognition #Cognitive psychology #Natural Language Processing Techniques #Psychiatry #Psychology

paper · doi:10.1038/s41562-024-02096-z

published in Nature Human Behaviour 9(2), 230-233 (Nature Portfolio)

openalex publication_date 2025/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Cited by

• Do AI Models Perform Human-like Abstract Reasoning Across Modalities?
• Large Language Models Pass the Turing Test
• Re-evaluating Theory of Mind evaluation in large language models
• Anthropocentric bias in language model evaluation
• The cognitive biases that may exacerbate inflationary and deflationary positions about large language models
• LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
• Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
• The fragility of "cultural tendencies" in LLMs
• Large Language Model probabilities cannot distinguish between possible and impossible language
• Large Language Models are Highly Aligned with Human Ratings of Emotional Stimuli
• How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs
• A validity-guided workflow for robust large language model research in psychology
• From Prompts to Constructs: A Dual-Validity Framework for LLM Research in Psychology
• UniCog: Uncovering Cognitive Abilities of LLMs through Latent Mind Space Analysis
• Human-Like Anaphor Resolution in Large Language Models
• A Suite of LMs Comprehend Puzzle Statements as Well or Better Than Humans. [europepmc]
• Large language models pass a standard three-party Turing test. [europepmc]

Discussions

• My commentary on the do's and don'ts of cognitive evaluations in LLMs is now out in Nature Human Behavior: doi.org/10.1038/s415... posting here with a figure that didn't make it into the final draf [bsky, 64 points, 0 comments]