2021/02/07 by Subhradeep Kayal, Kayal, Subhradeep
Computer Science · Mathematics · #Advanced Graph Neural Networks #Algorithm #Artificial intelligence #Atlas (anatomy) #Computation and Language (cs.CL) #Computer science #Encoder #FOS: Computer and information sciences #Geometry #Manifold (fluid mechanics) #Mathematics #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Projection (relational algebra) #Scalability #Scalar (mathematics) #Sentence #Theoretical computer science #Topological and Geometric Data Analysis #Unsupervised learning #Word (group theory) #cs.CL
paper · pdf · doi:10.48550/arxiv.2102.03795
Accepted at EACL 2021
arxiv created 2021/02/07 · openalex publication_date 2021/02/07 · arxiv updated 2021/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
The concept of unsupervised universal sentence encoders has gained traction\nrecently, wherein pre-trained models generate effective task-agnostic\nfixed-dimensional representations for phrases, sentences and paragraphs. Such\nmethods are of varying complexity, from simple weighted-averages of word\nvectors to complex language-models based on bidirectional transformers. In this\nwork we propose a novel technique to generate sentence-embeddings in an\nunsupervised fashion by projecting the sentences onto a fixed-dimensional\nmanifold with the objective of preserving local neighbourhoods in the original\nspace. To delineate such neighbourhoods we experiment with several set-distance\nmetrics, including the recently proposed Word Mover's distance, while the\nfixed-dimensional projection is achieved by employing a scalable and efficient\nmanifold approximation method rooted in topological data analysis. We test our\napproach, which we term EMAP or Embeddings by Manifold Approximation and\nProjection, on six publicly available text-classification datasets of varying\nsize and complexity. Empirical results show that our method consistently\nperforms similar to or better than several alternative state-of-the-art\napproaches.\n