Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis

2025/08/22 by Teo Sušnjak, Susnjak, Teo · 2 citations
Computer Science · Decision Sciences · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Meta-analysis and systematic reviews #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2509.00038

openalex publication_date 2025/08/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) offer significant potential to accelerate systematic literature reviews (SLRs), yet current approaches often rely on brittle, manually crafted prompts that compromise reliability and reproducibility. This fragility undermines scientific confidence in LLM-assisted evidence synthesis. In response, this work adapts recent advances in declarative prompt optimisation, developed for general-purpose LLM applications, and demonstrates their applicability to the domain of SLR automation. This research proposes a structured, domain-specific framework that embeds task declarations, test suites, and automated prompt tuning into a reproducible SLR workflow. These emerging methods are translated into a concrete blueprint with working code examples, enabling researchers to construct verifiable LLM pipelines that align with established principles of transparency and rigour in evidence synthesis. This is a novel application of such approaches to SLR pipelines.

Citations

• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
• What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction
• Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
• Benchmarking Prompt Sensitivity in Large Language Models
• Efficacy of Large Language Models in Systematic Reviews
• Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

Cited by

• Automated Risk-of-Bias Assessment of Randomized Controlled Trials: A First Look at a GEPA-trained Programmatic Prompting Framework
• Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification

Related

• Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing
• AI-Driven Scholarly Peer Review via Persistent Workflow Prompting, Meta-Prompting, and Meta-Reasoning
• Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification
• PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models
• PROMPTHEUS: A Human-Centered Pipeline to Streamline SLRs with LLMs