Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
2023/09/24 by R. Thomas McCoy, McCoy, R. Thomas, Shunyu Yao +7 · 13 voices · 20 citations
Computer Science · #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling #cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2309.13638
openalex publication_date 2023/09/24 · openalex created_date 2023/09/27 · openalex updated_date 2026/07/30
Abstract
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. We predict that LLMs will achieve higher accuracy when these probabilities are high than when they are low - even in deterministic settings where probability should not matter. To test our predictions, we evaluate two LLMs (GPT-3.5 and GPT-4) on eleven tasks, and we find robust evidence that LLMs are influenced by probability in the ways that we have hypothesized. In many cases, the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability word sequence but only 13% when it is low-probability. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system - one that has been shaped by its own particular set of pressures.
Cited by
Discussions
- arxiv.org/abs/2309.13638
I hope there are more papers like this in the future. [bsky, 4 points, 1 comments]
- Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
arxiv.org/abs/2309.13638 [bsky, 4 points, 0 comments]
- Understanding LLMs Through the Problem They Are Trained to Solve [hn, 4 points, 0 comments]
- This paper “embers of autoregression” is a brilliant riposte to the “sparks” paper, arguing reasoning capabilities displayed by LLMs are closer to “approximate retrieval” than resembling human intelli [bsky, 3 points, 2 comments]
- This paper presents a really clear explanation of things that LLM “AI” tools are and are not good at arxiv.org/abs/2309.13638 . It also contains one of my favorite kind of inline citations, to “(Sueto [bsky, 3 points, 1 comments]
- Understanding LLMs Through the Problem They Are Trained to Solve [hn, 2 points, 2 comments]
- I will look at this. But I have found the attached discussion to be compelling as to why LLMs do well on some tasks and do poorly on others. I think it fits well with Titus’ more philosophical analysi [bsky, 2 points, 1 comments]
- Just learned for the first time about this Princeton response to the infamously crummy MSFT puffpiece, "Sparks of AGI."
Haven't read it yet but the title alone is well worth the price of admission.
[bsky, 2 points, 0 comments]
- With some more prompting, it got to 2 columns (if you count a name as a word). That said, there's some reason to think this might be exactly the kind of task GPT would NOT excel at. Check out this pap [bsky, 1 points, 0 comments]
- Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
arxiv.org/abs/2309.13638 [bsky, 1 points, 0 comments]
- From here: arxiv.org/abs/2309.13638 [bsky, 0 points, 0 comments]
- A gem of a paper! Shows how the probability of text examples appearing in model training data impacts model performance. Compares "LLM-think" to "human-think"! Long read but worth your time!
Embers o [bsky, 0 points, 0 comments]
- These are cool papers on examining LLMs as statistical systems rather than reasoning system. They look at tasks and evaluate the models' sensitivity to probability even if the tasks themselves are det [bsky, 0 points, 0 comments]
Related