arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.25857cs.LG
收藏

基于LLM的上下文分子性质预测:关于记忆与知识冲突的盲化研究

In-Context Molecular Property Prediction with LLMs: A Blinding Study on Memorization and Knowledge Conflicts

Institute for Continuum and Material Mechanics, Technical University of Hamburg(持续与材料力学研究所,汉堡技术大学);Institute of Material Systems Modeling, Helmholtz-Zentrum Hereon(材料系统建模研究所,海德堡-这里恩研究中心);Institute of Surface Science, Helmholtz-Zentrum Hereon(表面科学研究所,海德堡-这里恩研究中心);German Center for Artificial Intelligence (DFKI), Saarland, Germany(德国人工智能中心(DFKI),萨尔兰州,德国);Saarland University(萨尔兰州大学)

Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Christian Feiler, Roland C. Aydin

AI总结:

通过渐进盲化实验和上下文样本量控制,研究LLM在分子性质预测中是否真正进行上下文回归,还是依赖记忆值,并分析预训练知识与上下文信息的冲突。

摘要:

大型语言模型(LLM)的能力已从自然语言处理扩展到科学预测任务,包括分子性质预测。然而,它们在上下文学习中的有效性仍然不明确,特别是在广泛使用的基准测试中存在训练数据污染的潜在可能。本文研究LLM是否对分子性质进行真正的上下文回归,还是主要依赖记忆值。此外,我们通过一系列渐进盲化实验,分析预训练知识与上下文信息之间的相互作用。我们使用系统盲化方法,逐步减少可用信息,评估三个系列(GPT-4.1、GPT-5、Gemini 2.5)中的九种LLM变体在三个MoleculeNet数据集(Delaney溶解度、亲脂性、QM7原子化能)上的表现。作为补充,我们利用不同的上下文样本量(0-shot、60-shot和1000-shot)作为信息访问的额外控制。这项工作为在受控信息访问下评估分子性质预测提供了一个原则性框架,解决了对记忆的担忧,并揭示了预训练知识与上下文信息之间的冲突。

英文摘要:

The capabilities of large language models (LLMs) have expanded beyond natural language processing to scientific prediction tasks, including molecular property prediction. However, their effectiveness in in-context learning remains ambiguous, particularly given the potential for training data contamination in widely used benchmarks. This paper investigates whether LLMs perform genuine in-context regression on molecular properties or instead rely on verbatim retrieval of memorized target values. Furthermore, we analyze the interplay between pre-trained knowledge and in-context information through a series of progressively blinded experiments. We evaluate nine LLM variants across three families (GPT-4.1, GPT-5, Gemini 2.5) on three MoleculeNet datasets (Delaney solubility, Lipophilicity, QM7 atomization energy) using a systematic blinding approach that iteratively reduces available information, complemented by 0-, 60-, and 1000-shot in-context sample sizes as an additional control for information access. To validate the memorization analysis and the blinding experiments, we add a positive and a negative control for the memorization experiments and structural reference baselines for the multi-shot experiments as well as bootstrap confidence intervals for all results. We find no evidence of verbatim retrieval on the legacy benchmarks and show that blinding exposes conflicts between pre-trained knowledge and in-context information. This work provides a principled framework for evaluating molecular property prediction under controlled information access.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑