Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by\n Averaging Source Word Embeddings

2018/04/14 by Joshua Coates, Coates, Joshua, Danushka Bollegala +1
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1804.05262

openalex publication_date 2018/04/14 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28

Abstract

Creating accurate meta-embeddings from pre-trained source embeddings has\nreceived attention lately. Methods based on global and locally-linear\ntransformation and concatenation have shown to produce accurate\nmeta-embeddings. In this paper, we show that the arithmetic mean of two\ndistinct word embedding sets yields a performant meta-embedding that is\ncomparable or better than more complex meta-embedding learning methods. The\nresult seems counter-intuitive given that vector spaces in different source\nembeddings are not comparable and cannot be simply averaged. We give insight\ninto why averaging can still produce accurate meta-embedding despite the\nincomparability of the source vector spaces.\n

Related

• Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings
• Learning Geometric Word Meta-Embeddings
• Think Globally, Embed Locally --- Locally Linear Meta-embedding of Words
• Angular-Based Word Meta-Embedding Learning
• Learning Meta-Embeddings by Using Ensembles of Embedding Sets