WebRetriever：用于高效网络智能体评估的大规模综合基准测试
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
AI总结：
针对现有网络智能体评估基准测试的局限，提出WebRetriever这一大规模综合基准测试，涵盖多领域网站和任务。采用NavEval框架及三个评估协议，实验揭示不同协议性能差异，为网络智能体研发提供细粒度见解和严谨基础。
摘要：
随着网络智能体在自动任务执行中的能力不断展现，开发强大的评估框架以评估其导航和任务完成性能成为关键研究重点。现有基准测试存在根本局限，如规模不足、领域多样性有限等。为此引入WebRetriever，它涵盖800个网站和1550个跨领域任务。提出NavEval框架，利用丰富交互上下文实现与人类判断高度对齐。还建立三个评估协议。实验表明各协议性能有差异，WebRetriever能为智能体能力提供细粒度诊断见解，推动网络智能体研发。
英文摘要：
As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
