arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12984cs.CL
收藏

SciAgentGym:LLM代理中多步科学工具使用的基准测试

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

Fudan NLP Group(复旦大学自然语言处理组)
Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zha… 展开作者

Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin, Tao Gui, Xingjun Ma, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang

AI总结:

为解决当前基准忽视代理在科学工作流中编排工具能力的问题,提出包含1780个领域特定工具的可扩展交互环境SciAgentGym和分层评估套件SciAgentBench,并设计数据合成方法SciForge,通过微调使SciAgent-8B超越更大模型,展示科学工具使用能力的跨领域正迁移。

摘要:

科学推理本质上需要整合复杂的工具包来导航领域特定知识。然而,当前的基准测试在很大程度上忽视了代理编排工具以进行此类严谨工作流的能力。为弥补这一差距,我们引入了SciAgentGym,一个可扩展的交互环境,包含四个自然科学学科中的1,780个领域特定工具,并由强大的执行基础设施支持。此外,我们提出了SciAgentBench,一个分层评估套件,旨在从基本动作到长期工作流对代理能力进行压力测试。我们的评估识别出一个关键瓶颈:最先进的模型仍然难以处理复杂的科学工具使用,并且随着交互范围的扩展,其性能显著下降。为解决这一问题,我们提出了SciForge,一种数据合成方法,将工具动作空间建模为依赖图,以生成逻辑感知的训练轨迹。通过在这些轨迹上进行微调,我们的SciAgent-8B优于显著更大的Qwen3-VL-235B-Instruct,同时表现出科学工具使用能力的跨领域正迁移。这些结果凸显了下一代自主科学代理的潜力。

英文摘要:

Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑