In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

2026/03/31 by Matthias Busch, Marius Tacke, Sviatlana V. Lamaka +4 · 2 citations
Computer Science · #cs.LG

paper · pdf · doi:10.48550/arxiv.2603.25857

arxiv created 2026/08/04 · arxiv updated 2026/08/05

Abstract

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multi-shot experiments as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pre-trained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

Citations

• A Survey on Data Contamination for Large Language Models
• How Much Can We Forget about Data Contamination?
• ChemLLM: A Chemical Large Language Model
• The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Lost in the Middle: How Language Models Use Long Contexts
• What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
• ChemCrow: Augmenting large-language models with chemistry tools
• What learning algorithm is in-context learning? Investigations with linear models
• Quantifying Memorization Across Neural Language Models
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Language Models are Few-Shot Learners
• Analyzing Learned Molecular Representations for Property Prediction
• Neural Message Passing for Quantum Chemistry
• MoleculeNet: A Benchmark for Molecular Machine Learning
• Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning
• ESOL: Estimating Aqueous Solubility Directly from Molecular Structure
• SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules