SIGIR 2025 -- LiveRAG Challenge Report

2025/07/07 by David Carmel, Simone Filice, Carmel, David +15
Computer Science · #Computation and Language (cs.CL) #Expert finding and Q&A systems #FOS: Computer and information sciences #H.3.3 #Information Retrieval (cs.IR) #Information Retrieval and Search Behavior #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.04942

openalex publication_date 2025/07/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The LiveRAG Challenge at SIGIR 2025, held between March and May 2025, provided a competitive platform for advancing Retrieval-Augmented Generation (RAG) technologies. Participants from academia and industry were invited to develop a RAG-based question-answering system using a fixed corpus (Fineweb-10BT) and a common open-source LLM (Falcon3-10B-Instruct). The goal was to facilitate challenging comparisons of retrieval and prompting strategies. During the Live Challenge Day, 70 teams from 27 different countries provided answers and supportive information to 500 unseen questions within a strict two-hour time window. Evaluation was conducted in two stages: first an automated LLM-as-a-judge approach was used to compute correctness and faithfulness score, then a manual review of top ranked submissions was conducted. The finalists were announced on June 12, 2025, with prizes awarded during the LiveRAG Workshop at SIGIR 2025 in Padua, Italy.

Citations

• HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval
• UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses
• Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation
• Knowledge-Aware Diverse Reranking for Cross-Source Question Answering
• RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering
• PreQRAG -- Classify and Rewrite for Enhanced RAG
• RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition
• LTRR: Learning To Rank Retrievers for LLMs
• DoTA-RAG: Dynamic of Thought Aggregation RAG
• CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training
• Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
• The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
• Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana
• A Survey on LLM-as-a-Judge
• Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
• The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• CRAG -- Comprehensive RAG Benchmark
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Design and Evaluation of a Retrieval-Augmented Generation Architecture for OWASP Security Data
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs
• The Fast Johnson–Lindenstrauss Transform and Approximate Nearest Neighbors

Related

• RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge
• RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition
• LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation
• TopClustRAG at SIGIR 2025 LiveRAG Challenge
• Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge