2015/10/09 by Benjamin J. Wilson, Wilson, Benjamin J., Adriaan M. J. Schakel +1
Computer Science · #68T50 #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #acm:68T50 #cs.CL #msc:68T50
paper · pdf · doi:10.48550/arxiv.1510.02675
Chagelog: Rerun experiment with subsampling turned off; re-interpreted results in light of Schnabel et al. (2015). 15 pages
arxiv created 2015/12/14 · arxiv updated 2015/12/15
An experimental approach to studying the properties of word embeddings is proposed. Controlled experiments, achieved through modifications of the training corpus, permit the demonstration of direct relations between word properties and word vector direction and length. The approach is demonstrated using the word2vec CBOW model with experiments that independently vary word frequency and word co-occurrence noise. The experiments reveal that word vector length depends more or less linearly on both word frequency and the level of noise in the co-occurrence distribution of the word. The coefficients of linearity depend upon the word. The special point in feature space, defined by the (artificial) word with pure noise in its co-occurrence distribution, is found to be small but non-zero.