2019/07/13 by Abhijit Mahalunkar, John D. Kelleher, Mahalunkar, Abhijit +1
Computer Science · Materials Science · #Advanced Graph Neural Networks #FOS: Computer and information sciences #Formal Languages and Automata Theory (cs.FL) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1907.06048
openalex publication_date 2019/07/13 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
In order to successfully model Long Distance Dependencies (LDDs) it is\nnecessary to understand the full-range of the characteristics of the LDDs\nexhibited in a target dataset. In this paper, we use Strictly k-Piecewise\nlanguages to generate datasets with various properties. We then compute the\ncharacteristics of the LDDs in these datasets using mutual information and\nanalyze the impact of factors such as (i) k, (ii) length of LDDs, (iii)\nvocabulary size, (iv) forbidden subsequences, and (v) dataset size. This\nanalysis reveal that the number of interacting elements in a dependency is an\nimportant characteristic of LDDs. This leads us to the challenge of modelling\nmulti-element long-distance dependencies. Our results suggest that attention\nmechanisms in neural networks may aide in modeling datasets with multi-element\nlong-distance dependencies. However, we conclude that there is a need to\ndevelop more efficient attention mechanisms to address this issue.\n