2025/05/24 by Daniel J. Korchinski, Dhruva Karkada, Korchinski, Daniel J. +5 · 1 voice · 1 citation
Computer Science · Physics and Astronomy · #Computation and Language (cs.CL) #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #cond-mat.dis-nn #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2505.18651
arxiv published 2025/05/24 · arxiv updated 2025/10/23
Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability P(i,j) of words i and j in text corpora. The resulting vectors Wi not only group semantically similar words but also exhibit a striking linear analogy structure -- for example, Wking - Wman + Wwoman ≈ Wqueen -- whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of the matrix M(i,j) = P(i,j)/P(i)P(j), (ii) strengthens and then saturates as more eigenvectors of M (i, j), which controls the dimension of the embeddings, are included, (iii) is enhanced when using log M(i,j) rather than M(i,j), and (iv) persists even when all word pairs involved in a specific analogy relation (e.g., king-queen, man-woman) are removed from the corpus. To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.