2023/10/02 by Aman Bhargava, Bhargava, Aman, Cameron Witkowski +4 · 9 citations
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Language and cultural evolution #Machine Learning (cs.LG) #Neural Networks and Reservoir Computing #Neural and Evolutionary Computing (cs.NE) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2310.04444
openalex publication_date 2023/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Prompt engineering is crucial for deploying LLMs but is poorly understood mathematically. We formalize LLM systems as a class of discrete stochastic dynamical systems to explore prompt engineering through the lens of control theory. We offer a mathematical analysis of the limitations on the controllability of self-attention as a function of the singular values of the parameter matrices. We present complementary empirical results on the controllability of a panel of LLMs, including Falcon-7b, Llama-7b, and Falcon-40b. Given initial state \mathbf x0 from Wikitext and prompts of length k ≤ 10 tokens, we find that the "correct" next token is reachable at least 97% of the time, and that the top 75 most likely next tokens are reachable at least 85% of the time. Intriguingly, short prompt sequences can dramatically alter the likelihood of specific outputs, even making the least likely tokens become the most likely ones. This control-theoretic analysis of LLMs demonstrates the significant and poorly understood role of input sequences in steering output probabilities, offering a foundational perspective for enhancing language model system capabilities.