2024/07/15 by Joos, Lucas, Keim, Daniel A., Fischer, Maximilian T. · 3 citations
#Digital Libraries (cs.DL) #FOS: Computer and information sciences #H.5.2 #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2407.10652
Systematic literature reviews (SLRs) are essential but labor-intensive due to high publication volumes and inefficient keyword-based filtering. To streamline this process, we evaluate Large Language Models (LLMs) for enhancing efficiency and accuracy in corpus filtration while minimizing manual effort. Our open-source tool LLMSurver presents a visual interface to utilize LLMs for literature filtration, evaluate the results, and refine queries in an interactive way. We assess the real-world performance of our approach in filtering over 8.3k articles during a recent survey construction, comparing results with human efforts. The findings show that recent LLM models can reduce filtering time from weeks to minutes. A consensus scheme ensures recall rates >98.8%, surpassing typical human error thresholds and improving selection accuracy. This work advances literature review methodologies and highlights the potential of responsible human-AI collaboration in academic research.