Into the Unknown: Benchmarking AI agents on open scientific research problems
Carnegie Mellon University(卡内基梅隆大学);University of Minnesota(明尼苏达大学);Stanford University(斯坦福大学);Trillion Labs;Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);Sandia National Laboratory(桑迪亚国家实验室);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Ames National Laboratory(艾姆斯国家实验室);Iowa State University(爱荷华州立大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Michigan State University(密歇根州立大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);TelePIX;Johns Hopkins University(约翰斯·霍普金斯大学);NEC Laboratories Europe(NEC欧洲实验室);Ss.Cyril and Methodius University of Skopje(斯科普里圣西里尔和圣美多德大学)
Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Gar… 展开作者
Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck
本研究推出Into the Unknown基准,测试9个前沿智能体撰写开放科学问题后续论文的能力,发现无智能体达到领域科学家标准,仍有较大提升空间。
摘要:
AI智能体能开展开放科学问题的研究吗?我们推出了Into the Unknown这一基准,它为智能体提供一篇科学论文以及一个要探索的研究方向,并要求智能体撰写一篇后续论文。我们与美国能源部(DOE)国家实验室的科学家合作,每位科学家都贡献了其自身研究中一个未发表的后续研究方向,以及投稿必须满足的必要条件。尽管总体而言评估后续论文的质量颇具挑战性,但对于没有参考答案的研究问题,利用这些条件形成评分标准可使评估更易处理。我们让9个前沿智能体各撰写15篇后续论文,涵盖量子计算、等离子体物理、计算化学、材料科学和科学机器学习领域。在135份投稿中,没有一份满足所有条件,114份未通过我们的完整性检查,最常见的原因是论文声称完成了从未开展的工作,或报告的数字与记录的证据相矛盾。总体而言,智能体能轻易产出看起来像后续论文的内容,但没有一份达到领域科学家的标准,仍有很大的提升空间。我们的代码和数据已公开可用。
英文摘要:
Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper along with a research direction to pursue and tasks it to write a follow-up paper. We work with scientists at US Department of Energy (DOE) national laboratories, who each contribute an unpublished follow-up direction from their own work together with the necessary conditions a submission must meet. While assessing the quality of a follow-up paper is challenging in general, using these conditions to form a rubric makes evaluation more tractable for research problems that have no reference answer. We have nine frontier agents write 15 follow-ups each, spanning quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning domains. None of the 135 submissions satisfies all the conditions, and 114 fail our integrity check, most often because the paper claims work that was never done or reports numbers that contradict the recorded evidence. Overall, agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist, leaving substantial headroom. Our code and data are publicly available.
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Yale University(耶鲁大学);Broad Institute of MIT and Harvard(布罗德研究所);The Pennsylvania State University(宾夕法尼亚州立大学);Northeastern University(东北大学);Northwestern University(西北大学)
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
Asta Team, Allen Institute for AI(Asta团队,人工智能研究所);University of Maryland(马里兰大学);University of Arizona(亚利桑那大学);University of Zurich(苏黎世大学);Bar-Ilan University(巴伊兰大学);University of Washington(华盛顿大学)
Rutherford Appleton Laboratory, Science and Technology Facilities Council(卢瑟福·阿普尔顿实验室,科学与技术设施委员会);Oak Ridge National Laboratory(橡树岭国家实验室);University of Virginia(弗吉尼亚大学)
Multiphysics Bench: Benchmarking and Investigating Scientific Machine Learning for Multiphysics PDEs
xLeaF Lab, Hong Kong University of Science and Technology (Guangzhou)(xLeaF实验室,香港科技大学(广州));National University of Singapore(新加坡国立大学);AI for Science, Shanghai AI Lab(AI for Science,上海人工智能实验室)
InnoGym: Benchmarking the Innovation Potential of AI Agents
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);National University of Singapore(新加坡国立大学);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(知识图谱联合实验室)
AI Directorate Fermilab(AI部门费米实验室);Biocomplexity Institute University of Virginia(生物复杂性研究所弗吉尼亚大学);CS Dept University of Wisconsin-Madison(计算机科学系威斯康星大学麦迪逊分校);Discovery Partners Institute University of Illinois Urbana-Champaign(发现伙伴研究所伊利诺伊大学厄巴纳-香槟分校);Biocomplexity Institute(生物复杂性研究所)
ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
Institute of Trustworthy Embodied AI(可信具身AI研究院);Shanghai Innovation Institution(上海创新机构);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身AI重点实验室);College of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);OpenMOSS;Wuhan University(武汉大学);Nanjing University(南京大学);Jilin University(吉林大学)
BLADE: Benchmarking Language Model Agents for Data-Driven Science
University of Washington(华盛顿大学);UC Berkeley(伯克利大学);New York University(纽约大学);Stanford University(斯坦福大学);University of British Columbia(不列颠哥伦比亚大学);Microsoft(微软);George Washington University(乔治·华盛顿大学)
Benchmarking energy consumption and latency for neuromorphic computing in condensed matter and particle physics
University of Twente(特文特大学);Radboud University(拉德堡德大学);IBM Research Europe - Zürich(IBM欧洲研究院-苏黎世);Nikhef(荷兰国家亚原子物理研究所);Eidgenössische Technische Hochschule Zürich(苏黎世联邦理工学院);SURF Cooperation(SURF合作组织);University of Valencia-CSIC(瓦伦西亚大学-西班牙国家研究委员会)
STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking
McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA, USA(麦戈文脑科学研究所,麻省理工学院,马萨诸塞州剑桥市);Fylo Labs Inc., New York, NY, USA(Fylo实验室公司,纽约州纽约市);Allen Institute for Brain Science, Seattle, WA, USA(艾伦脑科学研究所,华盛顿州西雅图市)
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学)
Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility
Shanghai Jiao Tong University(上海交通大学);OpenDataLab, Shanghai Artificial Intelligence Laboratory(OpenDataLab,上海人工智能实验室);The University of Hong Kong(香港大学);Peking University(北京大学)
SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Fudan University(复旦大学);Tsinghua University(清华大学);Beihang University(北航大学)
LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control
Shenzhen Loop Area Institute(深圳河套学院);Dalian University of Technology(大连理工大学);The Chinese University of Hong Kong(香港中文大学);The Hong Kong University of Science and Technology(香港科技大学)
ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);National University of Singapore(新加坡国立大学);Wuhan University(武汉大学)
Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
State Key Laboratory of Cognitive Intelligence(认知智能国家重点实验室);University of Science and Technology of China(中国科学技术大学);Artificial Intelligence Research Institute(人工智能研究院);iFLYTEK Co., Ltd(iFLYTEK公司)
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心科学智能创新数据实验室);Beihang University(北京航空航天大学)
ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Nanyang Technological University(南洋理工大学);Centre for Biomedical Data Science, Duke-NUS Medical School(杜克-新加坡国立大学医学院生物医学数据科学中心);Zhejiang University(浙江大学);Jurisprudence Research Association, China Law Society(中国法学会法理学研究会)
Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging
College of Future Technology and the National Biomedical Imaging Center, Peking University(北京大学未来技术学院和国家生物医学成像中心);AI for Science Institute (AISI)(科学人工智能研究所);University of Michigan(密歇根大学);State Key Laboratory of Acoustics and Marine Information, Institute of Acoustics, Chinese Academy of Sciences(中国科学院声学研究所声场声信息国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);School of Astronautics, Beihang University(北京航空航天大学宇航学院);Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education(教育部航天器设计优化与动态仿真技术重点实验室)
Sci$Π$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
University of Maryland, College Park(马里兰大学帕克分校);Gradient Network;Stanford University(斯坦福大学);The Chinese University of Hong Kong(香港中文大学);Dartmouth College(达特茅斯学院);University of California, Los Angeles(加州大学洛杉矶分校)
Benchmarking Autonomy in Scientific Experiments: A Hierarchical Taxonomy for Autonomous Large-Scale Facilities
University of Greenwich, Old Royal Naval College, Park Row, London, SE10 9LS, United Kingdom(格林威治大学,皇家海军学院,伦敦公园行,SE10 9LS,英国);ISIS Neutron & Muon Source, Rutherford Appleton Laboratory, Didcot, OX11 0QX, United Kingdom(ISIS中子与穆子源,拉瑟福德-苹果顿实验室,迪茨,OX11 0QX,英国);The Faraday Institution, Harwell Science and Innovation Campus, Didcot, OX11 0RA, United Kingdom(法拉第机构,哈威尔科学与创新校园,迪茨,OX11 0RA,英国)
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Baptist University(香港浸会大学);University of Oxford(牛津大学);New York University, Abu Dhabi(纽约大学阿布扎比分校)