InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents
Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Zhejiang University(浙江大学);University of Oxford(牛津大学);MemTensor (Shanghai) Technology Co., Ltd(MemTensor(上海)科技有限公司)
Information seeking is a fundamental requirement for humans. However, existing LLM agents rely heavily on open-web search, which exposes two fundamental weaknesses: online content is noisy and unreliable, and many real-world tasks require precise, domain-specific knowledge unavailable from the web. The emergence of the Model Context Protocol (MCP) now allows agents to interface with thousands of specialized tools, seemingly resolving this limitation. Yet it remains unclear whether agents can effectively leverage such tools -- and more importantly, whether they can integrate them with general-purpose search to solve complex tasks. Therefore, we introduce InfoMosaic-Bench, the first benchmark dedicated to multi-source information seeking in tool-augmented agents. Covering six representative domains (medicine, finance, maps, video, web, and multi-domain integration), InfoMosaic-Bench requires agents to combine general-purpose search with domain-specific tools. Tasks are synthesized with InfoMosaic-Flow, a scalable pipeline that grounds task conditions in verified tool outputs, enforces cross-source dependencies, and filters out shortcut cases solvable by trivial lookup. This design guarantees both reliability and non-triviality. Experiments with 14 state-of-the-art LLM agents reveal three findings: (i) web information alone is insufficient, with GPT-5 achieving only 38.2% accuracy and 67.5% pass rate; (ii) domain tools provide selective but inconsistent benefits, improving some domains while degrading others; and (iii) 22.4% of failures arise from incorrect tool usage or selection, highlighting that current LLMs still struggle with even basic tool handling.
Supervising the search process produces reliable and generalizable information-seeking agents
Department of Computer Science, University of Virginia, USA(弗吉尼亚大学计算机科学系);National Library of Medicine, National Institutes of Health, USA(美国国立卫生研究院国家医学图书馆);Department of Computer Science, University of Illinois Urbana–Champaign, USA(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Medical Oncology, Dana–Farber Cancer Institute, USA(达纳-法伯癌症研究所医学肿瘤科);Surgery, University of Alabama at Birmingham, USA(阿拉巴马大学伯明翰分校外科系);Department of Neurology, Yale School of Medicine, USA(耶鲁医学院神经病学系)
Uncovering Limitations of Large Language Models in Information Seeking from Tables
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)(中国科学院智能信息处理重点实验室);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Harbin Engineering University(哈尔滨工程大学);Peng Cheng Laboratory(鹏城实验室)
Seek Before You Move: Evidence Seeking for Progress Grounding in Vision-Language Navigation
Tsinghua University(清华大学);Pengcheng Laboratory(鹏城实验室);South China University of Technology(华南理工大学);International Digital Economy Academy(国际数字经济学院);Ping An Technology (Shenzhen) Co., Ltd.(平安科技(深圳)有限公司);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking
Renmin University of China(中国人民大学);Key Laboratory of Data Engineering and Knowledge Engineering, MOE(数据工程与知识工程重点实验室);Tsinghua University(清华大学);Institute of Education(教育学院);Department of Computer Science and Technology(计算机科学与技术系);Engineering Research Center of Database and Business Intelligence, MOE(数据库与商务智能工程研究中心);School of Information(信息学院)
针对学术查询中的复杂 API 耦合问题,提出基于预构建 API 调用方案与代码推理的 SoAy 方法,并通过 SoAyBench/SoAyEval 验证其显著优于现有基线。
DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
Pohang University of Science and Technology(釜山科学技术大学);The University of Sydney(悉尼大学);The University of Western Australia(西澳大学);The University of Melbourne(墨尔本大学)
FaithDial: A Faithful Benchmark for Information-Seeking Dialogue
University of Edinburgh(爱丁堡大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所);University of Alberta(阿尔伯塔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);IBM Research(IBM研究院);WeChat AI, Tencent(腾讯微信人工智能实验室)
HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks
Peking University(北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);University of Science and Technology of China(中国科学技术大学);The Hong Kong University of Science and Technology(香港科技大学)