Transformers, Contextualism, and Polysemy

2024/04/15 by Jumbly Grindrod, Grindrod, Jumbly · 2 citations
Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Law in Society and Culture

paper · pdf · doi:10.48550/arxiv.2404.09577

openalex publication_date 2024/04/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The transformer architecture, introduced by Vaswani et al. (2017), is at the heart of the remarkable recent progress in the development of language models, including widely-used chatbots such as Chat-GPT and Claude. In this paper, I argue that we can extract from the way the transformer architecture works a theory of the relationship between context and meaning. I call this the transformer theory, and I argue that it is novel with regard to two related philosophical debates: the contextualism debate regarding the extent of context-sensitivity across natural language, and the polysemy debate regarding how polysemy should be captured within an account of word meaning.

Cited by

• Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity
• Word Meanings in Transformer Language Models

Related

• Why transformers are obviously good models of language
• What Context Features Can Transformer Language Models Use?
• A Primer on the Inner Workings of Transformer-based Language Models
• From Counts to Meaning: Embeddings and Transformers
• What do language models model? Transformers, automata, and the format of thought