arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21143cs.AIcs.CLcs.IRcs.LG
收藏

深度信息合成的基准测试

A Benchmark for Deep Information Synthesis

Huawei Noah’s Ark Lab, UK(华为诺亚实验室,英国);Imperial College London(伦敦帝国理工学院);UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心);University of Zurich(苏黎世大学);University of Sheffield(谢菲尔德大学);University of Cambridge(剑桥大学)
Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan Alex S… 展开作者

Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang, Jun Wang, Gerasimos Lampouras

AI总结:

DEEPSYNTH是一个评估智能体在复杂任务中信息合成与推理能力的新基准测试,通过多阶段数据收集流程和120个跨7个领域的任务,评估LLM在现实问题中的表现。

摘要:

基于大型语言模型(LLM)的智能体越来越多地被用于解决涉及工具使用的复杂任务,如网络浏览、代码执行和数据分析。然而,当前的评估基准并不能充分评估其解决现实任务的能力,这些任务需要从多个来源合成信息并推断出超越简单事实检索的洞察。为此,我们引入了DEEPSYNTH,一个全新的基准测试,旨在评估智能体在现实、耗时的问题上的能力,这些问题结合了信息收集、合成和结构化推理以产生洞察。DEEPSYNTH包含120个任务,涵盖7个领域和数据源,覆盖67个国家。DEEPSYNTH使用多阶段的数据收集管道构建,要求标注者收集官方数据源、创建假设、进行手动分析并设计具有可验证答案的任务。在DEEPSYNTH上评估时,11种最先进的LLM和深度研究智能体在LLM-评估指标上达到了最高F1得分为8.97和17.5,突显了该基准测试的难度。我们的分析表明,当前智能体在幻觉和处理大规模信息空间的推理方面存在困难,突显了DEEPSYNTH作为指导未来研究的关键基准测试。

英文摘要:

Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑