从研究目标到科学闭环:通过证据支持的物理学研究评估智能体
From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research
AI总结:
该研究推出PhysClosureBench基准评估前沿LLM的科学调查能力,发现Claude-Opus-5局部科学成就最高但研究闭环能力不足,整体存在局部成就与完整闭环的差距。
摘要:
AI智能体能否组织能得出充分支持结论的科学调查?为评估这一能力,我们推出PhysClosureBench,这是一个包含50项经专家验证、目标明确的研究任务的基准,涵盖强场、阿秒及邻近物理学领域。每项任务明确研究目标和必要的科学约束,同时让智能体自行组织调查并确定如何生成和验证回答问题所需的证据。20项标准的原子分数衡量局部科学成就,而研究闭环(RC0-RC4)则根据不可或缺的证据要求和适用范围条件是否得到满足,评估调查在多大程度上解决了指定的研究问题。我们评估了7个前沿大语言模型(LLM),其中Claude-Opus-5的平均原子分数最高(6.29/10),但仅在32.7%的尝试中达到RC4。在整个基准中,36.2%的得分至少8/10的尝试仍未达到RC4,且所有7个模型在所有三轮运行中达到RC4的任务数量均少于至少某一轮的数量。案例级分析进一步表明,对生成结果的误解和未解决的物理适用范围限制会导致闭环不完整。这些结果共同揭示了AI智能体的局部科学成就与其实现完整研究闭环的能力之间存在持续的差距。
英文摘要:
Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。