arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.05080cs.CLcs.AIcs.LG
收藏

ScienceAgentBench:面向数据驱动科学发现的语言智能体严格评估

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, B… 展开作者

Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, Huan Sun

AI总结:

本文提出ScienceAgentBench基准,从44篇论文提取102个任务评估数据驱动科学发现的语言智能体,发现当前最强模型在专家知识辅助下仅解决34.3%任务,凸显其代码生成与端到端自动化能力的局限。

摘要:

大型语言模型(LLMs)的进步引发了人们对开发基于LLM的语言智能体以端到端自动化科学发现的日益浓厚的兴趣,这对其真实能力既带来了兴奋,也引发了怀疑。在这项工作中,我们呼吁在对端到端自动化做出大胆声明之前,先对科学工作流中的个体任务上的智能体进行严格评估。为此,我们提出了ScienceAgentBench,一个用于评估数据驱动科学发现语言智能体的新基准。为确保基准的科学真实性和现实相关性,我们从四个学科的44篇同行评审出版物中提取了102个任务,并邀请九位领域专家对其进行验证。我们将每个任务的目标输出统一为一个自包含的Python程序文件,并采用一系列评估指标来检查生成的程序、执行结果和成本。每个任务都经过标注员和领域专家的多轮手动验证,以确保其标注质量和科学合理性。我们还提出了两种有效策略来缓解数据污染问题。使用ScienceAgentBench,我们评估了五个开放权重和专有LLM,每个LLM均采用三种框架:直接提示、OpenHands CodeAct和自我调试。在为每个任务提供三次尝试的情况下,表现最好的智能体仅能独立解决32.4%的任务,在获得专家提供的知识后可解决34.3%。此外,我们评估了采用直接提示和自我调试的OpenAI o1-preview,它可将性能提升至42.2%,这证明了增加推理时计算的有效性,但其成本是其他LLM的10倍以上。尽管如此,我们的结果仍凸显了当前语言智能体在生成数据驱动发现代码方面的局限性,更不用说科学研究的端到端自动化了。

英文摘要:

The advancements of large language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about their true capabilities. In this work, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using ScienceAgentBench, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. In addition, we evaluate OpenAI o1-preview with direct prompting and self-debug, which can boost the performance to 42.2%, demonstrating the effectiveness of increasing inference-time compute but with more than 10 times the cost of other LLMs. Still, our results underscore the limitations of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑