2019/04/02 by Timo Schick, Hinrich Schütze, Schick, Timo +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1904.01617
openalex publication_date 2019/04/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Learning high-quality embeddings for rare words is a hard problem because of\nsparse context information. Mimicking (Pinter et al., 2017) has been proposed\nas a solution: given embeddings learned by a standard algorithm, a model is\nfirst trained to reproduce embeddings of frequent words from their surface form\nand then used to compute embeddings for rare words. In this paper, we introduce\nattentive mimicking: the mimicking model is given access not only to a word's\nsurface form, but also to all available contexts and learns to attend to the\nmost informative and reliable contexts for computing an embedding. In an\nevaluation on four tasks, we show that attentive mimicking outperforms previous\nwork for both rare and medium-frequency words. Thus, compared to previous work,\nattentive mimicking improves embeddings for a much larger part of the\nvocabulary, including the medium-frequency range.\n