arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04751cs.AI
收藏

在平行世界中评估搜索代理

Evaluating the Search Agent in a Parallel World

Li Auto

Jiawei Chen, Xintian Shen, Lihao Zheng, Lifu Mu, Haoyi Sun, Ning Mao, Hao Ma, Tao Wei, Pan Zhou, Kun Zhan

AI总结:

本文提出Mind-ParaWorld框架,通过生成未来场景和不可侵犯的原子事实,解决传统搜索代理评估中的基准构建、动态过时和证据判断难题。

摘要:

整合网络搜索工具显著扩展了LLMs处理开放世界、实时和长尾问题的能力。然而,评估这些搜索代理面临严峻挑战:构建高质量深度搜索基准成本过高,未验证的合成数据来源不可靠;静态基准面临动态过时:互联网信息变化导致复杂查询退化为简单检索,地面真实值因时间偏移而过时;归因模糊性干扰评估,因为代理性能常由参数记忆主导而非实际搜索和推理能力;依赖特定商业搜索引擎引入可变性,阻碍可重复性。为解决这些问题,我们提出Mind-ParaWorld框架,在平行世界中评估搜索代理。具体而言,MPW采样现实世界实体名称生成未来场景和问题,位于模型知识截止点之外。ParaWorld法律模型构建一组不可分割的原子事实和每个问题的唯一地面真实值。在评估过程中,代理与ParaWorld引擎模型互动,动态生成基于这些不可侵犯原子事实的搜索结果页面(SERPs)。我们发布了MPW-Bench,一个覆盖19个领域的交互式基准,包含1,608个实例。实验显示,尽管搜索代理在完整信息下擅长证据综合,其性能受限于在陌生搜索环境中的证据收集和覆盖范围,以及不可靠的证据充分性判断和何时停止决策瓶颈。

英文摘要:

Integrating web search tools has significantly extended the capability of LLMs to address open-world, real-time, and long-tail problems. However, evaluating these Search Agents presents formidable challenges. First, constructing high-quality deep search benchmarks is prohibitively expensive, while unverified synthetic data often suffers from unreliable sources. Second, static benchmarks face dynamic obsolescence: as internet information evolves, complex queries requiring deep research often degrade into simple retrieval tasks due to increased popularity, and ground truths become outdated due to temporal shifts. Third, attribution ambiguity confounds evaluation, as an agent's performance is often dominated by its parametric memory rather than its actual search and reasoning capabilities. Finally, reliance on specific commercial search engines introduces variability that hampers reproducibility. To address these issues, we propose a novel framework, Mind-ParaWorld, for evaluating Search Agents in a Parallel World. Specifically, MPW samples real-world entity names to synthesize future scenarios and questions situated beyond the model's knowledge cutoff. A ParaWorld Law Model then constructs a set of indivisible Atomic Facts and a unique ground-truth for each question. During evaluation, instead of retrieving real-world results, the agent interacts with a ParaWorld Engine Model that dynamically generates SERPs grounded in these inviolable Atomic Facts. We release MPW-Bench, an interactive benchmark spanning 19 domains with 1,608 instances. Experiments across three evaluation settings show that, while search agents are strong at evidence synthesis given complete information, their performance is limited not only by evidence collection and coverage in unfamiliar search environments, but also by unreliable evidence sufficiency judgment and when-to-stop decisions-bottlenecks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑