arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14189cs.CLcs.AI
收藏

知道何时不回答:具有退避意识的科学推理

Knowing When Not to Answer: Abstention-Aware Scientific Reasoning

Texas A&M University(德克萨斯农工大学);Hamad Bin Khalifa University(哈马德·本·哈利法大学)

Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

AI总结:

本文提出一种具有退避意识的科学推理框架,通过分解科学主张并利用自然语言推理评估证据,以减少错误风险,强调退避在科学验证中的重要性。

摘要:

大型语言模型越来越多地用于回答和验证科学主张,但现有的评估通常假设模型必须始终给出确定性的答案。然而,在科学领域,未经支持或不确定的结论可能比退避更具危害性。我们通过一种具有退避意识的验证框架来研究这个问题,该框架将科学主张分解为最小条件,利用自然语言推理(NLI)对每个条件进行审核,并选择性地决定是否支持、反驳或退避。我们评估了该框架在两个互补的科学基准上的表现:SciFact和PubMedQA,涵盖了闭书和开放领域证据设置。实验使用了六个不同的语言模型,包括编码器-解码器模型、开放式权重聊天模型和专有API。在所有基准和模型中,我们观察到原始准确率在架构之间变化很小,而退避在控制错误方面起着关键作用。特别是,基于置信度的退避在中等覆盖水平下显著降低了风险,即使绝对准确率的提升有限。我们的结果表明,在科学推理任务中,主要挑战不是选择单个最佳模型,而是确定可用证据是否足以证明一个答案。这项工作强调了具有退避意识的评估作为评估科学可靠性的实用且模型无关的视角,并为未来在科学领域进行选择性推理的研究提供了统一的实验基础。代码可在https://github.com/sabdaljalil2000/ai4science上获得。

英文摘要:

Large language models are increasingly used to answer and verify scientific claims, yet existing evaluations typically assume that a model must always produce a definitive answer. In scientific settings, however, unsupported or uncertain conclusions can be more harmful than abstaining. We study this problem through an abstention-aware verification framework that decomposes scientific claims into minimal conditions, audits each condition against available evidence using natural language inference (NLI), and selectively decides whether to support, refute, or abstain. We evaluate this framework across two complementary scientific benchmarks: SciFact and PubMedQA, covering both closed-book and open-domain evidence settings. Experiments are conducted with six diverse language models, including encoder-decoder, open-weight chat models, and proprietary APIs. Across all benchmarks and models, we observe that raw accuracy varies only modestly across architectures, while abstention plays a critical role in controlling error. In particular, confidence-based abstention substantially reduces risk at moderate coverage levels, even when absolute accuracy improvements are limited. Our results suggest that in scientific reasoning tasks, the primary challenge is not selecting a single best model, but rather determining when available evidence is sufficient to justify an answer. This work highlights abstention-aware evaluation as a practical and model-agnostic lens for assessing scientific reliability, and provides a unified experimental basis for future work on selective reasoning in scientific domains. Code is available at https://github.com/sabdaljalil2000/ai4science .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑