How to evaluate the cognitive abilities of LLMs

2025/01/15 by Anna A. Ivanova · 1 voice · 9 citations
Social Sciences · Computer Science · #Artificial Intelligence in Law #Natural Language Processing Techniques

paper · doi:10.1038/s41562-024-02096-z

Cited by

• Do AI Models Perform Human-like Abstract Reasoning Across Modalities?
• Large Language Models Pass the Turing Test
• Re-evaluating Theory of Mind evaluation in large language models
• Anthropocentric bias in language model evaluation
• The cognitive biases that may exacerbate inflationary and deflationary positions about large language models
• LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
• Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL
• The fragility of "cultural tendencies" in LLMs
• Large Language Model probabilities cannot distinguish between possible and impossible language
• Large Language Models are Highly Aligned with Human Ratings of Emotional Stimuli

Discussions

• My commentary on the do's and don'ts of cognitive evaluations in LLMs is now out in Nature Human Behavior: doi.org/10.1038/s415... posting here with a figure that didn't make it into the final draf [bsky, 64 points, 0 comments]