Had Enough of Experts? Quantitative Knowledge Retrieval From Large Language Models

2025/03/17 by David Selby, Yuichiro Iwashita, Kai Spriestersbach +6 · 1 voice
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Expert finding and Q&A systems #Topic Modeling

paper · pdf · doi:10.1002/sta4.70054

openalex publication_date 2025/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/09

Abstract

ABSTRACT Large language models (LLMs) have been extensively studied for their ability to generate convincing natural language sequences; however, their utility for quantitative information retrieval is less well understood. Here, we explore the feasibility of LLMs as a mechanism for quantitative knowledge retrieval to aid two data analysis tasks: elicitation of prior distributions for Bayesian models and imputation of missing data. We introduce a framework that leverages LLMs to enhance Bayesian workflows by eliciting expert‐like prior knowledge and imputing missing data. Tested on diverse datasets, this approach can improve predictive accuracy and reduce data requirements, offering significant potential in healthcare, environmental science and engineering applications. We discuss the implications and challenges of treating LLMs as ‘experts’.

Citations

• The Prior Can Often Only Be Understood in the Context of the Likelihood
• Probabilistic supervised learning
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• OpenAGI: When LLM Meets Domain Experts
• Solving Math Word Problems by Combining Language Models With Symbolic Solvers
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• Task Contamination: Language Models May Not Be Few-Shot Anymore
• AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling
• Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Discussions

• Just published: 'Had enough of experts? Quantitative retrieval from large language models' Can LLMs, having read the scientific literature, offer us useful numerical info to help fill in missing data [bsky, 1 points, 0 comments]

Related

• Measuring and Modifying Factual Knowledge in Large Language Models
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
• LLM Inference Enhanced by External Knowledge: A Survey
• Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization