RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

2023/09/26 by Ronak Pradeep, Pradeep, Ronak, Sahel Sharifymoghaddam +3 · 11 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2309.15088

openalex publication_date 2023/09/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. To address this significant shortcoming, we present RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. Experimental results on the TREC 2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter model, although our effectiveness remains slightly behind reranking with GPT-4. We hope our work provides the foundation for future research on reranking with modern LLMs. All the code necessary to reproduce our results is available at https://github.com/castorini/rank_llm.

Cited by

• SciClaimSeekers at CheckThat! 2026: Retrieving Scientific Sources for Social Media Claims with LLM Reranking
• The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Models
• What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models
• E-CARE: An Efficient LLM-based Commonsense-Augmented Framework for E-Commerce
• Embedding-Based Context-Aware Reranker
• Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short
• Scalable In-context Ranking with Generative Models
• Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs
• jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking
• Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
• Access Paths for Efficient Ordering with Large Language Models

Related

• RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!
• LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking
• Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion
• Zero-Shot Listwise Document Reranking with a Large Language Model
• Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models