The emergent algebraic structure of RNNs and embeddings in NLP

2018/03/07 by Cantrell, Sean A.
#97R40 #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1803.02839

Abstract

We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.

Related

• The geometry of integration in text classification RNNs
• Character n-gram Embeddings to Improve RNN Language Models
• Return of the RNN: Residual Recurrent Networks for Invertible Sentence Embeddings
• Supervised Understanding of Word Embeddings
• Encoding word order in complex embeddings