arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.18449cs.AI
收藏

自主科学基准:评估用于自主科学发现的AI智能体

Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery

Northwestern University(西北大学)

Yibo Wen, Han Liu

AI总结:

本研究推出自主科学基准Autonomous Science Bench,含6项多学科预算实验流程,经试点评估发现不同智能体在发现与学习关卡存在差异化缺陷,为自主科学发现智能体提供更精准评估工具。

摘要:

用于自主科学发现的AI智能体必须在有限预算下决定开展哪些实验。大多数科学智能体基准提供固定数据集,而其他支持主动实验的基准则很大程度上依赖模拟。我们推出Autonomous Science Bench(自主科学基准),其初始版本包含6项预算约束下的设计-构建-测试-学习(design-build-test-learn)流程,这些流程基于生物学、化学和材料科学领域已发表的实验。智能体选择批量设计,从实验室服务获取测量结果,并利用这些结果指导后续实验,最终生成预测模型。成功需通过两个校准关卡:发现关卡评估智能体测量的设计,学习关卡评估其模型对保留设计的排名。在对8种智能体-模型配置和144次试验的试点评估中,最佳配置在67%的试验中成功,而最难的流程仅在24次尝试中成功3次。两个关卡揭示了不同的缺陷:部分智能体构建了准确的排名,但发现的有前景设计过少;另一些智能体发现了有前景设计,但无法预测哪些未见过的设计会表现良好。这些差异仅通过整体成功率会被掩盖。我们在该httpsURL发布了流程、实验室运行时和公开排行榜的0.1版本。

英文摘要:

AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide fixed datasets, while others that support active experimentation largely rely on simulation. We introduce Autonomous Science Bench, an initial release comprising six budgeted design-build-test-learn campaigns grounded in published experiments across biology, chemistry, and materials science. Agents select batches of designs, receive measurements from a lab service, and use the results to guide subsequent experiments before delivering a predictive model. Success requires passing two calibrated gates: a discovery gate, which evaluates the designs the agent measured, and a learning gate, which evaluates its model's ranking of held-out designs. In a pilot evaluation of ten agent-model configurations and 180 trials, the best configuration succeeds in 72% of trials, while the hardest campaign is solved in only five of 30 attempts. The two gates reveal different weaknesses: some agents build accurate rankings but find too few promising designs, while others find promising designs but fail to predict which unseen designs will perform well. These differences are obscured by overall success rates alone. We release version 0.1 of the campaigns, lab runtime, and public leaderboard at https://asb.open-ssi.org/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑