AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents
FAIR at Meta(Meta 的 FAIR 部门);University of Oxford(牛津大学);University College London(伦敦大学学院)
Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipo… 展开作者
Alisia Lupidi, Bhavul Gauri, Thomas Simon Foster, Bassel Al Omari, Despoina Magka, Alberto Pepe, Alexis Audran-Reiss, Muna Aghamelu, Nicolas Baldwin, Lucia Cipolina-Kun, Jean-Christophe Gagnon-Audet, Chee Hau Leow, Sandra Lefdal, Hossam Mossalam, Abhinav Moudgil, Saba Nazir, Emanuel Tewolde, Isabel Urrego, Jordi Armengol Estape, Amar Budhiraja, Gaurav Chaurasia, Abhishek Charnalia, Derek Dunfield, Karen Hambardzumyan, Daniel Izcovich, Martin Josifoski, Ishita Mediratta, Kelvin Niu, Parth Pathak, Michael Shvartsman, Edan Toledo, Anton Protopopov, Roberta Raileanu, Alexander Miller, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach
LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
MIT(麻省理工学院);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);University of California, Los Angeles(加州大学洛杉矶分校);University of California, San Diego(加州大学圣地亚哥分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Michigan(密歇根大学);National University of Singapore(新加坡国立大学);University of Tokyo(东京大学)
REVEAL: Reverse Engineering Agentic Latents from Frontier-Model Traces
Zhejiang University(浙江大学);South China University of Technology(华南理工大学);Alibaba Group(阿里巴巴集团);Fudan University(复旦大学);City University of Hong Kong(香港城市大学)
Your AI Travel Agent Would Book You a Bullfight: An Agentic Benchmark for Implicit Animal Welfare in Frontier AI Models
Compassion Aligned Machine Learning(同情对齐机器学习);Sentient Futures(感知未来);Harvard Kennedy School(哈佛肯尼迪学院);Appalachian State University Department of Management(阿巴拉契亚州立大学管理系)
Evaluating Frontier LLMs on PhD-Level Mathematical Reasoning: A Benchmark on a Textbook in Theoretical Computer Science about Randomized Algorithms
Wyoming Seminary(怀俄明神学院);San Jose State University(圣何塞州立大学);Guilin University of Electronic Technology(桂林电子科技大学);University of California, Berkeley(加州大学伯克利分校);Northeastern University(东北大学);Guangdong University of Technology(广东工业大学)
Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
University of Texas at Austin(得克萨斯大学奥斯汀分校);The Ohio State University(俄亥俄州立大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达);Northwestern University(西北大学);Steadyworks
GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Universidad Católica de Ávila (UCAV)(阿维拉天主教大学);Johns Hopkins University(约翰霍普金斯大学);Independent Researcher(独立研究者);Center for Geographic Analysis, Harvard University(哈佛大学地理分析中心)
Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
The Wharton School, University of Pennsylvania(宾夕法尼亚大学沃顿商学院);Carnegie Mellon University(卡内基梅隆大学);Harvard Business School, Harvard University(哈佛大学哈佛商学院)
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
MATS;Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Astra Fellowship;ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center;LawZero;Google DeepMind(谷歌DeepMind)
Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming
Patronus AI University of California, Berkeley(Patronus AI 伯克利大学)
ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AI
Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Beijing Key Laboratory of Safe AI and Superalignment(北京安全人工智能与超对齐重点实验室);BrainCog Lab, Institute of Automation, Chinese Academy of Sciences(脑认知实验室,中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Long-term AI(长期人工智能)
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
School of Artificial Intelligence(人工智能学院);Zhiyuan College(智远学院);School of Physics and Astronomy(物理天文学院);Tsung-Dao Lee Institute(李政道研究所);State Key Laboratory of Dark Matter Physics(暗物质物理国家重点实验室);Shanghai Innovation Institute(上海创新研究院);Institute of Theoretical Physics(理论物理研究所);Chinese Academy of Sciences(中国科学院);Zhejiang Lab(浙江实验室);SciLand;DP Technology(DP技术)
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Department of Cybersecurity(网络安全系);Department of Information Sciences and Technology(信息科学与技术系);College of Emergency Preparedness, Homeland Security, and Cybersecurity(应急准备、国土安全与网络安全学院);University at Albany, SUNY(阿尔巴尼大学)