arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2611.06910cs.AI
收藏

FrontierPhysics:评估端到端物理学研究智能体的基准

FrontierPhysics: Evaluating Agents for End-to-End Physics Research

University of California, Berkeley(加州大学伯克利分校);California Institute of Technology(加州理工学院);Harvard University(哈佛大学);University of Washington(华盛顿大学);Stanford University(斯坦福大学);Amazon(亚马逊公司);University of Pittsburgh(匹兹堡大学);University of California, San Diego(加州大学圣地亚哥分校);ZEISS(蔡司公司);Georgia Institute of Technology(佐治亚理工学院);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);The Scripps Research Institute(斯克里普斯研究所);Oak Ridge National Laboratory(橡树岭国家实验室);University of Texas at Austin(德克萨斯大学奥斯汀分校);Purdue University(普渡大学);University of Toronto(多伦多大学);University of Southern California(南加州大学);Ludwig-Maximilians-Universität München(慕尼黑大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Davis(加州大学戴维斯分校);Princeton University(普林斯顿大学);Northeastern University(东北大学);University of Pennsylvania(宾夕法尼亚大学);University of Minnesota, Twin Cities(明尼苏达大学双城分校);New York University(纽约大学);Duke University(杜克大学);Nanyang Technological University(南洋理工大学);University of Palermo(巴勒莫大学);IBM(国际商业机器公司);University of Amsterdam(阿姆斯特丹大学);University of California, Santa Cruz(加州大学圣克鲁兹分校);Prime Intellect
Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma, Jiechao Feng, Bo Chen, Xiangkai Sun, Jiarui Liu, Jinen Guo, Zeyu Zhou, Xu… 展开作者

Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma, Jiechao Feng, Bo Chen, Xiangkai Sun, Jiarui Liu, Jinen Guo, Zeyu Zhou, Xu Huang, Zhuohan Li, Haoyu Li, Heguang Lin, Lijie Ding, Yadong Zeng, Kangqi Zhang, Xinge Liu, Hanwen Liu, Zhenghong Li, Alexander von Recum, Yuchen Zhu, Felix Schwarzfischer, Yizhou Zhao, Jicheng Wang, Binxu Li, Qian Yu, Chan Li, Cheng Dai, Di Wang, Haoyu Zhang, Jiankai Sun, Kaituo Feng, Mian Qin, Ruihong Yin, Yiqun Luo, Weiwei Wu, Shihong Pan, Tsung Yeh Hsieh, Xiao Huang, Yan Zhang, Sihan Li, Zirui Ren, Zengji Tu, Kyoung Whan Choe, Ke Sun, Vincenzo Galia, Vaisakhi Mishra, Jiashu He, Hongyang Wang, Zhiheng Yang, Shubham Gaur, Yifan Gu, Shuyi Wang, Zihao Zuo, Steven Dillmann, Yiyou Sun, Xinyang Han, Hanchung Lee, Molei Tao, Yue Zhao, Shilong Liu, Yuchen Wu, Xuandong Zhao, Xiangyi Li, Florian Brand, Gerbrand Ceder, Hartmut Haeffner

AI总结:

FrontierPhysics是覆盖9个物理领域的端到端智能体评估基准,含56项任务,结合验证器与专家评分标准评估智能体研究,发现前沿智能体存在推理与直觉短板。

摘要:

AI智能体在物理学研究中发挥着日益重要的作用。然而,现有的物理学基准仅评估固定问题、孤立子任务或带有给定解决方案的论文复现,很少同时检验研究质量与正确性。我们推出FrontierPhysics,这是一个由42位物理学家基于自身研究构建的包含56项任务的端到端基准,覆盖理论、计算和实验物理学的9个领域。每项任务要求智能体完成文献综述、制定研究计划、开展研究与实验,并生成结构化输出及科学手稿。结合确定性验证器与专家制定的评分标准,FrontierPhysics可评估智能体的研究是否正确且具备发表价值。在22种模型的26种配置中,没有任何配置能在168次试跑中通过超过46次。对智能体轨迹的人工审查显示,前沿智能体的主要弱点是推理严谨性不足和专家直觉有限,而非物理学知识或执行错误存在缺口。

英文摘要:

AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑