Core Knowledge Deficits in Multi-Modal Language Models

2024/10/06 by Yijiang Li, Li, Yijiang, Qingying Gao +19 · 2 voices · 7 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.2410.10855

openalex publication_date 2024/10/06 · openalex created_date 2024/10/20 · openalex updated_date 2026/07/28

Abstract

While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge--rudimentary cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale.

Cited by

• BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models
• Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules
• Multimodal Evaluation of Russian-language Architectures
• SRNN: Spatiotemporal Relational Neural Network for Intuitive Physics Understanding
• Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matrices
• Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling
• Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning

Discussions

• Sam is 100% correct on this. Indeed, human babies have essential cognitive priors such as permanence, continuity, and boundary of objects, 3D Euclidean understanding of space, etc. We spent 2 years to [bsky, 21 points, 0 comments]
• "Multi-modal LLMs do not genuinely advance toward core knowledge but instead rely on illusory understanding and shortcut learning as they scale." arxiv.org/abs/2410.10855 [bsky, 0 points, 0 comments]

Related

• The Philosophical Foundations of Growing AI Like A Child
• Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study
• Revealing the structure of language model capabilities
• Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
• On the Unexpected Abilities of Large Language Models