arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.18786cs.CLcs.AI
收藏

实验还是结果?在大型语言模型中探测科学可行性

Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models

University of Maryland, Baltimore County(马里兰大学巴尔的摩分校)

Seyedali Mohammadi, Manas Gaur, Francis Ferraro

AI总结:

本文探讨了在大型语言模型中科学可行性的评估,发现提供结果证据比实验描述更可靠,结果能提升准确性,而实验文本可能因上下文不完整而退化。

摘要:

科学可行性评估询问一个主张是否与已知知识一致,以及实验证据能否支持或反驳它。我们将可行性评估视为一种诊断推理任务,在给定假设的情况下,模型预测可行或不可行并给出理由。我们评估了在受控知识条件下(仅假设、有实验、有结果或两者都有)的大型语言模型(LLMs),并通过逐步移除实验和/或结果上下文来探测鲁棒性。在多个LLMs和两个数据集中,提供结果证据通常比提供实验描述更可靠。结果倾向于在内部知识之外提升准确性,而实验文本可能脆弱,当上下文不完整时性能会下降。这些发现澄清了在LLM基于可行性评估中实验证据何时有益,何时引入脆弱性。

英文摘要:

Scientific feasibility assessment asks whether a claim is consistent with established knowledge and whether experimental evidence could support or refute it. We frame feasibility assessment as a diagnostic reasoning task in which, given a hypothesis, a model predicts feasible or infeasible and justifies its decision. We evaluate large language models (LLMs) under controlled knowledge conditions (hypothesis-only, with experiments, with outcomes, or both) and probe robustness by progressively removing portions of the experimental and/or outcome context. Across multiple LLMs and two datasets, providing outcome evidence is generally more reliable than providing experiment descriptions. Outcomes tend to improve accuracy beyond what internal knowledge alone provides, whereas experimental text can be brittle and may degrade performance when the context is incomplete. These findings clarify when experimental evidence benefits LLM-based feasibility assessment and when it introduces fragility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑