A Simple Explanation for the Phase Transition in Large Language Models with List Decoding

2023/03/23 by Cheng‐Shang Chang, Chang, Cheng-Shang
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (stat.ML) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2303.13112

openalex publication_date 2023/03/23 · openalex created_date 2023/03/25 · openalex updated_date 2026/07/28

Abstract

Various recent experimental results show that large language models (LLM) exhibit emergent abilities that are not present in small models. System performance is greatly improved after passing a certain critical threshold of scale. In this letter, we provide a simple explanation for such a phase transition phenomenon. For this, we model an LLM as a sequence-to-sequence random function. Instead of using instant generation at each step, we use a list decoder that keeps a list of candidate sequences at each step and defers the generation of the output sequence at the end. We show that there is a critical threshold such that the expected number of erroneous candidate sequences remains bounded when an LLM is below the threshold, and it grows exponentially when an LLM is above the threshold. Such a threshold is related to the basic reproduction number in a contagious disease.

Related

• Phase transition in large language models and the criticality of natural languages
• First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models
• Predicting Emergent Abilities with Infinite Resolution Evaluation
• An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models
• Large Language Models as Markov Chains