The advancements of large language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about their true capabilities. In this work, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using ScienceAgentBench, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. In addition, we evaluate OpenAI o1-preview with direct prompting and self-debug, which can boost the performance to 42.2%, demonstrating the effectiveness of increasing inference-time compute but with more than 10 times the cost of other LLMs. Still, our results underscore the limitations of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists
Department of Computer Science and Engineering(计算机科学与工程系);College of Pharmacy(药学院);Department of Psychology(心理学系);Department of Biomedical Informatics(生物医学信息学系);Department of Geography(地理系);Department of Chemistry(化学系);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);The Ohio State University(俄亥俄州立大学)
Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);Fudan University(复旦大学);CUHK(香港中文大学);Tsinghua University(清华大学);CUHK-Shenzhen(香港中文大学深圳校区);UESTC(电子科技大学);USTC(中国科学技术大学)
MMSciBench: Benchmarking Language Models on Chinese Multimodal Scientific Problems
Department of Computer Science, Yale University(耶鲁大学计算机科学系);Department of Computer Science, Brown University(布朗大学计算机科学系);School of Data Science, Fudan University(复旦大学数据科学学院);Shanghai Key Laboratory of Data Science(上海数据科学关键实验室);Datawiz LLC
Decoding complexity: how machine learning is redefining scientific discovery
Alice Author Affiliation(Alice作者机构);Tom Theory Affiliation(Tom理论机构);Emma Experiments Affiliation(Emma实验机构);Affiliation, Department, City, Country(机构、部门、城市、国家);Simon Simulations Affiliation(Simon模拟机构)
AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing
Department of Physics, Massachusetts Institute of Technology(麻省理工学院物理系);The NSF AI Institute for Artificial Intelligence and Fundamental Interactions(国家科学基金会人工智能与基本相互作用研究所);Department of Physics, University of Wisconsin, Madison(威斯康星大学麦迪逊分校物理系)
Theory-guided hard constraint projection (HCP): a knowledge-based data-driven scientific machine learning method
Peng Cheng Laboratory(鹏城实验室);The University of Tokyo(东京大学);Southern University of Science and Technology(南方科技大学);Peking University(北京大学);Malardalen University(马拉达伦大学);LocationMind Inc.(LocationMind公司)
Unsupervised Machine Learning for Scientific Discovery: Workflow and Best Practices
Department of Neuroscience, Baylor College of Medicine(神经科学系,贝勒医学院);Department of Applied and Computational Mathematics and Statistics, University of Notre Dame(应用与计算数学和统计学系,圣母大学);Department of Statistics, Columbia University(统计学系,哥伦比亚大学);Zuckerman Institute & Irving Institute, Columbia University(Zuckerman研究所与Irving研究所,哥伦比亚大学)
Accelerating scientific discovery with Co-Scientist
Google Cloud AI Research(谷歌云人工智能研究);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究);Stanford University School of Medicine(斯坦福大学医学院);Houston Methodist(休斯顿卫理公会医院);Sequome;Fleming Initiative and Imperial College London(弗莱明倡议与伦敦帝国理工学院)
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
Asta Team, Allen Institute for AI(Asta团队,人工智能研究所);University of Maryland(马里兰大学);University of Arizona(亚利桑那大学);University of Zurich(苏黎世大学);Bar-Ilan University(巴伊兰大学);University of Washington(华盛顿大学)
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学)
SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks
Duke University(杜克大学);Zhejiang University(浙江大学);Australian National University(澳大利亚国立大学);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);Beijing University of Posts and Telecommunications(北京邮电大学);University College London(伦敦大学学院);University of Glasgow(格拉斯哥大学);China Mobile Communications Group Co.,Ltd(中国移动通信集团有限公司);Hong Kong Polytechnic University(香港理工大学);North China University of Technology(北方工业大学)
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Peking University(北京大学);Nanjing University(南京大学);East China Normal University(华东师范大学);Yale University(耶鲁大学)
AI-Assisted Scientific Assessment: A Case Study on Climate Change
Potsdam Institute for Climate Impact Research (PIK), Member of the Leibniz Association(波茨坦气候影响研究所(PIK),莱比锡协会成员);Institute for Atmospheric and Climate Science, ETH Zurich(大气与气候科学研究所,苏黎世联邦理工学院);University of Zurich(苏黎世大学);University of Miami(迈阿密大学);Centre for Environmental Policy, Imperial College London(伦敦帝国理工学院环境政策中心);Grantham Institute for Climate Change and Environment, Imperial College London(伦敦帝国理工学院气候变化与环境研究所);Energy, Climate and Environment Program, International Institute for Applied Systems Analysis(国际应用系统分析研究所能源、气候与环境项目)
BIOME-Bench: A Benchmark for Biomolecular Interaction Inference and Multi-Omics Pathway Mechanism Elucidation from Scientific Literature
Beijing Wenge Technology Co., Ltd(北京文格科技有限公司);Experimental Research Center, China Academy of Chinese Medical Sciences(中国中医科学院实验研究中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学院);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东科学院));School of New Media and Communication, Tianjin University(天津大学新闻与传播学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)