arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06855cs.AI
收藏

AIRS-Bench: 一个面向前沿人工智能研究科学代理的任务集

AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents

FAIR at Meta(Meta 的 FAIR 部门);University of Oxford(牛津大学);University College London(伦敦大学学院)
Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipo… 展开作者

Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Chaurasia, Abhishek Charnalia, Derek Dunfield, Karen Hambardzumyan, Daniel Izcovich, Martin Josifoski, Ishita Mediratta, Kelvin Niu, Parth Pathak, Michael Shvartsman, Edan Toledo, Anton Protopopov, Roberta Raileanu, Alexander Miller, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach

AI总结:

AIRS-Bench通过20个任务评估代理在科研全生命周期中的能力,发现代理在部分任务中超越人类但未达理论上限,推动自主科研发展。

摘要:

大语言模型代理在推动科学研究方面具有重大潜力。为了加速这一进展,我们引入了AIRS-Bench(人工智能研究科学基准),这是一个包含20个任务的任务集,这些任务来源于最先进的机器学习论文。这些任务涵盖语言模型、数学、生物信息学和时间序列预测等多个领域。AIRS-Bench任务评估代理在整个研究生命周期中的能力,包括想法生成、实验分析和迭代完善,而不提供基准代码。AIRS-Bench任务格式具有灵活性,能够方便地集成新任务并在不同代理框架之间进行严格比较。我们使用前沿模型配以顺序和并行支架建立了基准。我们的结果表明,代理在四个任务中超过人类SOTA,但在其他十六个任务中未能达到人类水平。即使当代理超越人类基准时,它们也未能达到底层任务的理论性能上限。这些发现表明,AIRS-Bench远未饱和,仍具有显著的改进空间。我们开源了AIRS-Bench任务定义和评估代码,以促进自主科学研究的进一步发展。

英文摘要:

LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑