arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.25342cs.LG
收藏

从意图到证据:一种基于范畴论的深度研究代理结构评估方法

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

The Hong Kong University of Science and Technology(香港科学与技术大学);E Fund Management Co., Ltd.(E基金管理有限公司);The Hong Kong Polytechnic University(香港理工大学)

Shuoling Liu, Zhiquan Tan, Kun Yi, Hui Wu, Yihan Li, Jiangpeng Yan, Liyuan Chen, Kai Chen, Qiang Yang

AI总结:

本文提出一种基于范畴论的框架,用于评估和改进深度研究代理,通过结构化映射从用户意图到证据结论,提出296个双语问题基准,评估16个前沿系统,发现结构任务仍具挑战性,且提供实际系统改进方法。

摘要:

深度研究代理(DRAs)旨在通过搜索网络、检查证据并跨异构源综合结论来回答复杂问题。我们引入了一个范畴论框架,用于评估和改进此类代理。该框架将深度研究视为从用户意图到证据支持结论的结构化映射,使检索轨迹、跨源对齐和最终综合显式化。基于此观点,我们推导出一个包含296个双语问题的机制感知基准。该基准针对四个关键的现实研究技能:遵循多跳证据链、跨源验证主张、重新排列碎片化信息以及拒绝不支持的假设。我们评估了16个前沿系统,并通过人工验证发现这些结构任务仍然极具挑战性:最佳系统仅达到19.9%的平均准确率。结果表明,强大代理有时能重新组织证据并检测假前提,但仍难以处理长视界综合和高交集验证。除了评估,相同的理论还导致实际系统改进。我们实例化了理论指导的干预措施,如跟踪搜索(保留检索轨迹)和范畴工具(添加显式验证和综合步骤)。这些干预措施在API基于的深度研究系统中带来了可测量的提升。我们的工作因此为构建更可靠的代理提供了具有挑战性的基准和具体的指导。

英文摘要:

Deep Research Agents (DRAs) aim to answer complex questions by searching the web, checking evidence, and synthesizing conclusions across heterogeneous sources. We introduce a category-theoretic framework for evaluating and improving such agents. The framework treats deep research as a structured mapping from user intent to evidence-grounded conclusions, making retrieval traces, cross-source alignment, and final synthesis explicit. Guided by this view, we derive a mechanism-aware benchmark of 296 bilingual questions. The benchmark targets four structural skills central to real research: following multi-hop evidence chains, verifying claims across sources, re-ordering fragmented information, and rejecting unsupported assumptions. We evaluate 16 frontier systems with human verification and find that these structural tasks remain highly challenging: the best system reaches only 19.9% average accuracy. The results show that strong agents can sometimes reorganize evidence and detect false premises, but still struggle with long-horizon synthesis and intersection-heavy verification. Beyond evaluation, the same theory also leads to practical system improvements. We instantiate theory-guided interventions such as tracked search, which preserves retrieval traces, and category tools, which add explicit verification and synthesis steps. These interventions yield measurable gains in API-based deep research systems. Our work therefore provides both a challenging benchmark and concrete design guidance for building more reliable research agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑