Static Word Embeddings for Sentence Semantic Representation

2025/06/05 by Takashi Wada, Wada, Takashi, Yuki Hirakawa +7
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.04624

openalex publication_date 2025/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose new static word embeddings optimised for sentence semantic representation. We first extract word embeddings from a pre-trained Sentence Transformer, and improve them with sentence-level principal component analysis, followed by either knowledge distillation or contrastive learning. During inference, we represent sentences by simply averaging word embeddings, which requires little computational cost. We evaluate models on both monolingual and cross-lingual tasks and show that our model substantially outperforms existing static models on sentence semantic tasks, and even surpasses a basic Sentence Transformer model (SimCSE) on a text embedding benchmark. Lastly, we perform a variety of analyses and show that our method successfully removes word embedding components that are not highly relevant to sentence semantics, and adjusts the vector norms based on the influence of words on sentence semantics.

Citations

• mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
• Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
• It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
• Language-agnostic BERT Sentence Embedding
• On the Language Neutrality of Pre-trained Multilingual Representations
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• JParaCrawl: A Large Scale Web-Based English-Japanese Parallel Corpus
• CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB
• Unsupervised Cross-lingual Representation Learning at Scale
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• How Contextual are Contextualized Word Representations? Comparing the\n Geometry of BERT, ELMo, and GPT-2 Embeddings
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• Advances in Pre-Training Distributed Word Representations
• Word Translation Without Parallel Data
• Attention Is All You Need
• All-but-the-Top: Simple and Effective Postprocessing for Word Representations
• Enriching Word Vectors with Subword Information
• Distilling the Knowledge in a Neural Network
• Adam: A Method for Stochastic Optimization
• Exploiting Similarities among Languages for Machine Translation
• Efficient Estimation of Word Representations in Vector Space
• Long Short-Term Memory
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Paraphrase Thought: Sentence Embedding Module Imitating Human Language Recognition
• Sentence transition matrix: An efficient approach that preserves sentence semantics
• Semantic Sentence Embeddings for Paraphrasing and Text Summarization
• Optimal synthesis embeddings