2020/10/09 by Justin Dieter, Dieter, Justin, Arun Tejasvi Chaganty +1
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #I.2.6 #I.2.7 #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2010.04842
14 pages, 5 figures
arxiv created 2020/10/09 · openalex publication_date 2020/10/09 · arxiv updated 2020/10/13 · openalex created_date 2020/10/15 · openalex updated_date 2026/07/28
Pretrained (language) embeddings are versatile, task-agnostic feature representations of entities, like words, that are central to many machine learning applications. These representations can be enriched through retrofitting, a class of methods that incorporate task-specific domain knowledge encoded as a graph over a subset of these entities. However, existing retrofitting algorithms face two limitations: they overfit the observed graph by failing to represent relationships with missing entities; and they underfit the observed graph by only learning embeddings in Euclidean manifolds, which cannot faithfully represent even simple tree-structured or cyclic graphs. We address these problems with two key contributions: (i) we propose a novel regularizer, a conformality regularizer, that preserves local geometry from the pretrained embeddings---enabling generalization to missing entities and (ii) a new Riemannian feedforward layer that learns to map pre-trained embeddings onto a non-Euclidean manifold that can better represent the entire graph. Through experiments on WordNet, we demonstrate that the conformality regularizer prevents even existing (Euclidean-only) methods from overfitting on link prediction for missing entities, and---together with the Riemannian feedforward layer---learns non-Euclidean embeddings that outperform them.