Longformer for MS MARCO Document Re-ranking Task

2020/09/20 by Ivan Sekulić, Sekulić, Ivan, Amir Soleimani +5
Computer Science · #Advanced Text Analysis Techniques #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2009.09392

openalex publication_date 2020/09/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Two step document ranking, where the initial retrieval is done by a classical information retrieval method, followed by neural re-ranking model, is the new standard. The best performance is achieved by using transformer-based models as re-rankers, e.g., BERT. We employ Longformer, a BERT-like model for long documents, on the MS MARCO document re-ranking task. The complete code used for training the model can be found on: https://github.com/isekulic/longformer-marco

Citations

• Adam: A Method for Stochastic Optimization
• Passage Re-ranking with BERT
• Transformers: State-of-the-Art Natural Language Processing
• Longformer: The Long-Document Transformer

Related

• Learning-to-Rank with BERT in TF-Ranking
• How Different are Pre-trained Transformers for Text Ranking?
• SDR: Efficient Neural Re-ranking using Succinct Document Representation
• Pretrained Transformers for Text Ranking: BERT and Beyond
• RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers