On the Curious Case of ℓ₂ norm of Sense Embeddings

2022/10/26 by Zhou, Yi, Bollegala, Danushka
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2210.14815

Abstract

We show that the ℓ₂ norm of a static sense embedding encodes information related to the frequency of that sense in the training corpus used to learn the sense embeddings. This finding can be seen as an extension of a previously known relationship for word embeddings to sense embeddings. Our experimental results show that, in spite of its simplicity, the ℓ₂ norm of sense embeddings is a surprisingly effective feature for several word sense related tasks such as (a) most frequent sense prediction, (b) Word-in-Context (WiC), and (c) Word Sense Disambiguation (WSD). In particular, by simply including the ℓ₂ norm of a sense embedding as a feature in a classifier, we show that we can improve WiC and WSD methods that use static sense embeddings.

Related

• Incremental Sense Weight Training for the Interpretation of Contextualized Word Embeddings
• Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings
• Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
• Learning Word Sense Embeddings from Word Sense Definitions
• Sense Embedding Learning for Word Sense Induction