Large Language Models and Emergence: A Complex Systems Perspective

2025/06/10 by David C. Krakauer, John W. Krakauer, Krakauer, David C. +3 · 13 voices · 5 citations
#cs.CL #cs.AI #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2506.11135

Abstract

Emergence is a concept in complexity science that describes how many-body systems manifest novel higher-level properties, properties that can be described by replacing high-dimensional mechanisms with lower-dimensional effective variables and theories. This is captured by the idea "more is different". Intelligence is a consummate emergent property manifesting increasingly efficient -- cheaper and faster -- uses of emergent capabilities to solve problems. This is captured by the idea "less is more". In this paper, we first examine claims that Large Language Models exhibit emergent capabilities, reviewing several approaches to quantifying emergence, and secondly ask whether LLMs possess emergent intelligence.

Citations

• What Emergence Can Possibly Mean
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• Evaluating the World Model Implicit in a Generative Model
• Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
• Are Emergent Abilities in Large Language Models just In-Context Learning?
• The chemical basis of morphogenesis
• Are Emergent Abilities of Large Language Models a Mirage?

Cited by

• If LLMs Have Human-Like Attributes, Then So Does Age of Empires II
• Symmetries at the origin of hierarchical emergence
• Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interaction
• Relative-Based Scaling Law for Neural Language Models
• What is emergence, after all?

Discussions

• New paper: "Large Language Models and Emergence: A Complex Systems Perspective" (D. Krakauer, J. Krakauer, M. Mitchell). We look at claims of "emergent capabilities" & "emergent intelligence" in LLMs [bsky, 235 points, 7 comments]
• With perfect timing, @melaniemitchell.bsky.social and colleagues have just put out a paper that brings some of these issues into focus, not least why there are no good grounds for supposing we are jus [bsky, 15 points, 1 comments]
• Starting to read this because emergence means things (I took dynamical systems), but they're losing me with this distinction between capabilities and intelligence. I don't buy it. [bsky, 2 points, 1 comments]
• very interesting paper contrasting what is called "emergence" in LLMs with emergence as understood in complex systems theory arxiv.org/pdf/2506.11135 [bsky, 2 points, 0 comments]
• Large Language Models and Emergence: A Complex Systems Perspective [hn, 2 points, 0 comments]
• Large Language Models and Emergence: A Complex Systems Perspective [hn, 1 points, 0 comments]
• Our points about "emergence" are also echoed in recent work by Krakauer et al. (arxiv.org/pdf/2506.111...) [bsky, 1 points, 1 comments]
• Fascinating article from the SFI on emergence ad LLMs David C. Krakauer, John W. Krakauer, Melanie Mitchell arxiv.org/abs/2506.11135 #AI #LLM #emergence #SFI [bsky, 1 points, 0 comments]
• Sein aktuelles Paper "Large Language Models and Emergence: A Complex Systems Perspective" (arxiv.org/abs/2506.11135) untersucht Emergenz als ein Konzept aus der Komplexitätswissenschaft und insbesonde [bsky, 1 points, 1 comments]
• > This is captured by the idea "less is more". In this paper, we first examine claims that Large Language Models exhibit emergent capabilities, reviewing several approaches to quantifying emergence, a [bsky, 1 points, 0 comments]
• The question of emergent properties in LLMs is tricky, and the following paper presents the various claims and their objections. If I understood it correctly, it makes a distinction between emergent c [bsky, 0 points, 0 comments]
• arxiv.org/pdf/2506.11135 This paper from this year look into LLM emergence. It differentiates between 'emergent properties' - being able to identify the outline of an object in an image background and [bsky, 0 points, 1 comments]
• Large Language Models and Emergence: A Complex Systems Perspective (Krakauer, Krakauer, Mitchell) https://arxiv.org/pdf/2506.11135 [bsky, 0 points, 0 comments]

Related

• Emergent Abilities of Large Language Models
• Why are LLMs' abilities emergent?
• Emergent Abilities in Large Language Models: A Survey
• Are Emergent Abilities of Large Language Models a Mirage?
• Are Emergent Abilities in Large Language Models just In-Context Learning?