arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.02271cs.CLcs.AI
收藏

InfoMosaic-Bench:评估工具增强智能体中的多源信息寻求能力

InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents

Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);University of Oxford(牛津大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)科技有限公司)

Yaxin Du, Yuanshuo Zhang, Xiyuan Yang, Yifan Zhou, Cheng Wang, Gongyi Zou, Xianghe Pang, Wenhao Wang, Menglan Chen, Shuo Tang, Zhiyu Li, Feiyu Xiong, Siheng Chen

AI总结:

该研究提出首个面向工具增强智能体多源信息寻求的基准InfoMosaic-Bench,覆盖六大领域,通过可扩展流水线合成可靠非平凡任务,实验发现现有LLM智能体存在网络信息不足、领域工具增益不稳、工具操作能力欠缺等问题。

摘要:

信息寻求是人类的一项基本需求。然而,现有大语言模型(LLM)智能体严重依赖开放网络搜索,这暴露出两个根本缺陷:在线内容嘈杂且不可靠,同时许多现实世界任务需要网络无法提供的精准领域特定知识。模型上下文协议(Model Context Protocol,MCP)的出现如今让智能体能够与数千种专用工具对接,看似解决了这一局限。但目前尚不清楚智能体能否有效利用这类工具——更重要的是,它们能否将这类工具与通用搜索相结合以解决复杂任务。\n为此,我们提出InfoMosaic-Bench,这是首个专门针对工具增强智能体多源信息寻求的基准测试。该基准覆盖六个代表性领域(医学、金融、地图、视频、网络以及多领域整合),要求智能体将通用搜索与领域特定工具结合使用。任务通过InfoMosaic-Flow合成,这是一个可扩展的流水线,能够将任务条件建立在经过验证的工具输出之上,强制设置跨源依赖关系,并过滤掉仅靠简单查找就能解决的捷径案例,这种设计同时保证了可靠性和非平凡性。\n我们对14个最先进的LLM智能体开展了实验,得出三项发现:(i)仅靠网络信息是不够的,GPT-5的准确率仅为38.2%,通过率为67.5%;(ii)领域工具能带来选择性但不稳定的增益,在部分领域表现提升,在另一些领域却出现性能下降;(iii)22.4%的失败源于工具使用或选择错误,这表明当前LLM即便在基础工具操作方面仍存在困难。

英文摘要:

Information seeking is a fundamental requirement for humans. However, existing LLM agents rely heavily on open-web search, which exposes two fundamental weaknesses: online content is noisy and unreliable, and many real-world tasks require precise, domain-specific knowledge unavailable from the web. The emergence of the Model Context Protocol (MCP) now allows agents to interface with thousands of specialized tools, seemingly resolving this limitation. Yet it remains unclear whether agents can effectively leverage such tools -- and more importantly, whether they can integrate them with general-purpose search to solve complex tasks. Therefore, we introduce InfoMosaic-Bench, the first benchmark dedicated to multi-source information seeking in tool-augmented agents. Covering six representative domains (medicine, finance, maps, video, web, and multi-domain integration), InfoMosaic-Bench requires agents to combine general-purpose search with domain-specific tools. Tasks are synthesized with InfoMosaic-Flow, a scalable pipeline that grounds task conditions in verified tool outputs, enforces cross-source dependencies, and filters out shortcut cases solvable by trivial lookup. This design guarantees both reliability and non-triviality. Experiments with 14 state-of-the-art LLM agents reveal three findings: (i) web information alone is insufficient, with GPT-5 achieving only 38.2% accuracy and 67.5% pass rate; (ii) domain tools provide selective but inconsistent benefits, improving some domains while degrading others; and (iii) 22.4% of failures arise from incorrect tool usage or selection, highlighting that current LLMs still struggle with even basic tool handling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑