Enhancing Sentence Embedding with Generalized Pooling

2018/06/26 by Qian Chen, Chen, Qian, Zhen-Hua Ling +3 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1806.09828

openalex publication_date 2018/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.

Citations

• Skip-Thought Vectors
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Sentences and Documents
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering

Cited by

• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• Conditional Self-Attention for Query-based Summarization

Related

• Why and when should you pool? Analyzing Pooling in Recurrent\n Architectures
• Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?
• Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks
• Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models
• Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective