Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval

2024/11/07 by Ferdinand Schlatt, Maik Fröbe, Matthias Hagen · 1 voice
Computer Science · Decision Sciences · #Advanced Text Analysis Techniques #Data Quality and Management #Topic Modeling #cs.IR

paper · pdf · doi:10.1145/3701551.3704118

openalex publication_date 2025/02/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

A wide range of transformer-based language models have been proposed for information retrieval tasks. However, including transformer-based models in retrieval pipelines is often complex and requires substantial engineering effort. In this paper, we introduce Lightning IR, an easy-to-use PyTorch Lightning-based framework for applying transformer-based language models in retrieval scenarios. Lightning IR provides a modular and extensible architecture that supports all stages of a retrieval pipeline: from fine-tuning and indexing to searching and re-ranking. Designed to be scalable and reproducible, Lightning IR is available as open-source: https://github.com/webis-de/lightning-ir.

Citations

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Transformers: State-of-the-Art Natural Language Processing

Discussions

• New preprint of WSDM demo by @maik_froebe @matthias and Ferdinand Schlatt Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval https: [bsky, 9 points, 0 comments]

Related

• Large Language Models for Information Retrieval: A Survey
• Synergistic Interplay between Search and Large Language Models for Information Retrieval
• Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges
• A Survey of Model Architectures in Information Retrieval
• Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration