The emergent algebraic structure of RNNs and embeddings in NLP

2018/03/07 by Sean A. Cantrell, Cantrell, Sean A.
Computer Science · Mathematics · #97R40 #Algebraic number #Artificial Intelligence (cs.AI) #Artificial intelligence #Artificial neural network #Class (philosophy) #Combinatorics #Computation and Language (cs.CL) #Computer science #Dimension (graph theory) #Embedding #FOS: Computer and information sciences #Hyperparameter #Machine Learning (stat.ML) #Mathematics #Natural Language Processing Techniques #Natural language processing #Neural Networks and Applications #Pattern recognition (psychology) #Recurrent neural network #Representation (politics) #Topic Modeling #Word (group theory) #Word embedding #cs.AI #cs.CL #msc:97R40 #stat.ML

paper · pdf · doi:10.48550/arxiv.1803.02839

published in arXiv (Cornell University) (Cornell University) · 24 pages, 16 figures

arxiv created 2018/03/07 · openalex publication_date 2018/03/07 · arxiv updated 2018/03/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.

Citations

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Finding Structure in Time
• Gradient-based learning applied to document recognition
• Character-level Convolutional Networks for Text Classification
• Benefits of depth in neural networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• Comparative Study of CNN and RNN for Natural Language Processing
• Learning to Compute Word Embeddings On the Fly
• Cortical microcircuits as gated-recurrent neural networks
• StarSpace: Embed All The Things!
• Monotonic Chunkwise Attention

Related

• The geometry of integration in text classification RNNs
• Character n-gram Embeddings to Improve RNN Language Models
• Return of the RNN: Residual Recurrent Networks for Invertible Sentence Embeddings
• Supervised Understanding of Word Embeddings
• Encoding word order in complex embeddings