2020/07/10 by Xuan Shan, Shan, Xuan, Chuanjie Liu +15 · 1 citation
Computer Science · #Artificial intelligence #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Information retrieval #Key (lock) #Machine learning #Margin (machine learning) #Matching (statistics) #Multimodal Machine Learning Applications #Natural language processing #Search engine #Semantic search #Topic Modeling #Word (group theory) #cs.IR
paper · pdf · doi:10.48550/arxiv.2007.05186
published in arXiv (Cornell University) (Cornell University) · 8pages,2figures
openalex publication_date 2020/07/10 · arxiv created 2021/05/24 · arxiv updated 2021/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06
Deep matching models aim to facilitate search engines retrieving more relevant documents by mapping queries and documents into semantic vectors in the first-stage retrieval. When leveraging BERT as the deep matching model, the attention score across two words are solely built upon local contextualized word embeddings. It lacks prior global knowledge to distinguish the importance of different words, which has been proved to play a critical role in information retrieval tasks. In addition to this, BERT only performs attention across sub-words tokens which weakens whole word attention representation. We propose a novel Global Weighted Self-Attention (GLOW) network for web document search. GLOW fuses global corpus statistics into the deep matching model. By adding prior weights into attention generation from global information, like BM25, GLOW successfully learns weighted attention scores jointly with query matrix Q and key matrix K. We also present an efficient whole word weight sharing solution to bring prior whole word knowledge into sub-words level attention. It aids Transformer to learn whole word level attention. To make our models applicable to complicated web search scenarios, we introduce combined fields representation to accommodate documents with multiple fields even with variable number of instances. We demonstrate GLOW is more efficient to capture the topical and semantic representation both in queries and documents. Intrinsic evaluation and experiments conducted on public data sets reveal GLOW to be a general framework for document retrieve task. It significantly outperforms BERT and other competitive baselines by a large margin while retaining the same model complexity with BERT.