Benchmarking generative AI tools for literature retrieval and summarization in genomic variant interpretation

2025/10/01 by Andrea Gazzo, Silvia Berardelli, Matteo Biancospino +6 · 1 voice
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Topic Modeling #Natural Language Processing Techniques

paper · pdf · doi:10.1101/2025.09.29.679212

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/14

Abstract

Abstract Background Generative AI is increasingly used to extract structured information across domains, but its reliability in academic and clinical research, where precision and accuracy are essential, remains largely unexplored. This study evaluates the ability of Large Language Models (LLMs)-based algorithms to generate accurate, literature-based summaries of human genomic variants, with a focus on real-world usability. Results We benchmarked five open-access generative AI platforms—ChatGPT, MistralAI, VarChat, Perplexity, and ScholarAI—across 40 curated variants equally divided between somatic and germline settings. For each variant, summary reports were generated and blindly evaluated by domain experts using five defined metrics. VarChat emerged as the top-ranked tool, showing the highest summarization accuracy, citation relevance, and robustness against hallucinations. Gpt-4o consistently ranked second, showing particularly stable robustness in conditions where the literature was scarce. Perplexity and ScholarAI, despite being literature-focused, ranked lowest across most metrics. Tool performance was strongly influenced by the availability of peer-reviewed literature, confirming that current generative models remain sensitive to data scarcity. Conclusions Our findings highlight the heterogeneity of current generative AI tools in genomic variant interpretation workflows. While some platforms already provide useful outputs, reliable integration into basic and clinical research requires expert validation and domain-related fine-tuning. This work provides for the first time a curated benchmark for assessing LLM-generated content in variant genomics and underscores the need for caution when using these tools to support variant interpretation.

Citations

• Adapted large language models can outperform medical experts in clinical text summarization
• GPT-4o System Card
• DeepSeek-V3 Technical Report
• LitSumm: large language models for literature summarization of noncoding RNAs
• No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
• LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
• A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Archive of PubTator 3.0 source code and trained models
• LitSumm: Large language models for literature summarisation of non-coding RNAs
• Can Large Language Models Be an Alternative to Human Evaluations?
• tmVar 3.0: an improved variant concept recognition and normalization tool
• DeepVar: An End-to-End Deep Learning Approach for Genomic Variant Recognition in Biomedical Literature
• Attention Is All You Need

Discussions

• Benchmarking generative AI tools for literature retrieval and summarization in genomic variant interpretation doi.org/10.1101/2025... NAVIP: Unraveling the Influence of Neighboring Small Sequence Var [bsky, 0 points, 0 comments]