arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19159cs.AIcs.CLcs.LG
收藏

探索未知:在开放科学研究问题上对AI智能体进行基准测试

Into the Unknown: Benchmarking AI agents on open scientific research problems

Carnegie Mellon University(卡内基梅隆大学);University of Minnesota(明尼苏达大学);Stanford University(斯坦福大学);Trillion Labs;Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Sandia National Laboratory(桑迪亚国家实验室);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Ames National Laboratory(艾姆斯国家实验室);Iowa State University(爱荷华州立大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Michigan State University(密歇根州立大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);TelePIX;Johns Hopkins University(约翰斯·霍普金斯大学);NEC Laboratories Europe(NEC欧洲实验室);Ss.Cyril and Methodius University of Skopje(斯科普里圣西里尔和圣美多德大学)
Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Gar… 展开作者

Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck

AI总结:

本研究推出Into the Unknown基准,测试9个前沿智能体撰写开放科学问题后续论文的能力,发现无智能体达到领域科学家标准,仍有较大提升空间。

摘要:

AI智能体能开展开放科学问题的研究吗?我们推出了Into the Unknown这一基准,它为智能体提供一篇科学论文以及一个要探索的研究方向,并要求智能体撰写一篇后续论文。我们与美国能源部(DOE)国家实验室的科学家合作,每位科学家都贡献了其自身研究中一个未发表的后续研究方向,以及投稿必须满足的必要条件。尽管总体而言评估后续论文的质量颇具挑战性,但对于没有参考答案的研究问题,利用这些条件形成评分标准可使评估更易处理。我们让9个前沿智能体各撰写15篇后续论文,涵盖量子计算、等离子体物理、计算化学、材料科学和科学机器学习领域。在135份投稿中,没有一份满足所有条件,114份未通过我们的完整性检查,最常见的原因是论文声称完成了从未开展的工作,或报告的数字与记录的证据相矛盾。总体而言,智能体能轻易产出看起来像后续论文的内容,但没有一份达到领域科学家的标准,仍有很大的提升空间。我们的代码和数据已公开可用。

英文摘要:

Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper along with a research direction to pursue and tasks it to write a follow-up paper. We work with scientists at US Department of Energy (DOE) national laboratories, who each contribute an unpublished follow-up direction from their own work together with the necessary conditions a submission must meet. While assessing the quality of a follow-up paper is challenging in general, using these conditions to form a rubric makes evaluation more tractable for research problems that have no reference answer. We have nine frontier agents write 15 follow-ups each, spanning quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning domains. None of the 135 submissions satisfies all the conditions, and 114 fail our integrity check, most often because the paper claims work that was never done or reports numbers that contradict the recorded evidence. Overall, agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist, leaving substantial headroom. Our code and data are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑