2016/04/28 by Ehsaneddin Asgari, Asgari, Ehsaneddin, Mohammad R. K. Mofrad +1
Biochemistry, Genetics and Molecular Biology · Computer Science · #Authorship Attribution and Profiling #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning in Bioinformatics #Natural Language Processing Techniques
paper · pdf · doi:10.48550/arxiv.1604.08561
openalex publication_date 2016/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We introduce a new measure of distance between languages based on word\nembedding, called word embedding language divergence (WELD). WELD is defined as\ndivergence between unified similarity distribution of words between languages.\nUsing such a measure, we perform language comparison for fifty natural\nlanguages and twelve genetic languages. Our natural language dataset is a\ncollection of sentence-aligned parallel corpora from bible translations for\nfifty languages spanning a variety of language families. Although we use\nparallel corpora, which guarantees having the same content in all languages,\ninterestingly in many cases languages within the same family cluster together.\nIn addition to natural languages, we perform language comparison for the coding\nregions in the genomes of 12 different organisms (4 plants, 6 animals, and two\nhuman subjects). Our result confirms a significant high-level difference in the\ngenetic language model of humans/animals versus plants. The proposed method is\na step toward defining a quantitative measure of similarity between languages,\nwith applications in languages classification, genre identification, dialect\nidentification, and evaluation of translations.\n