arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.10909cs.AI
收藏

PaperArena: 一个用于科学文献上工具增强代理推理的评估基准

PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literature

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Artificial Intelligence Research Institute, iFLYTEK Co., Ltd(人工智能研究院,iFLYTEK有限公司)

Daoyu Wang, Mingyue Cheng, Shuo Yu, Zirui Liu, Ze Guo, Xin Li, Qi Liu

AI总结:

PaperArena提出一个评估基准,用于评估基于LLM的代理在跨多篇论文整合信息并使用外部工具进行推理的能力,实验显示现有代理在复杂任务上的表现有限。

摘要:

理解并推理大规模科学文献是基于大语言模型(LLM)的代理的重要考验。然而,现有工作主要局限于单篇论文内的工具自由任务,这主要是由于缺乏一个能够评估跨论文推理和多工具协调的基准,在真实研究场景中。在本工作中,我们提出了PaperArena,一个用于评估LLM代理在需要整合多篇论文信息并借助外部工具的问题上的基准。给定一个研究问题,代理应制定推理计划,与多篇论文互动,并调用适当的工具以生成有根据的答案。为了支持标准化评估,我们提供了一个代理执行平台,提供了一个模块化的工具环境,包括多模态解析、上下文检索和程序计算。实验表明,即使领先的LLM驱动的已建立的代理工作流也仅能达到38.78%的平均准确率,而在困难子集上,准确率降至仅18.47%。我们还分析了推理轨迹并诊断代理行为,为社区提供了见解,以开发和评估更强大的科学代理。

英文摘要:

Understanding and reasoning on the large-scale scientific literature is a crucial touchstone for large language model (LLM) based agents. However, existing works are mainly restricted to tool-free tasks within single papers, largely due to the lack of a benchmark that evaluates cross-paper reasoning and multi-tool orchestration in authentic research scenarios. In this work, we propose PaperArena, a benchmark to evaluate LLM-based agents on questions that require integrating information across multiple papers with the assistance of external tools. Given a research question, agents should formulate a reasoning plan, interact with multiple papers, and invoke appropriate tools to produce a well-grounded answer. To support standardized evaluation, we provide a platform for agent execution, offering a modular tool environment including multimodal parsing, context retrieval, and programmatic computation. Experiments reveal that even the leading LLM powering a well-established agentic workflow achieves merely 38.78% average accuracy, while on the hard subset, accuracy drops to only 18.47%. We also analyze reasoning traces and diagnose agent behavior, providing the community with insights to develop and evaluate more capable scientific agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑