arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.29020cs.AI
收藏

WebVoyager的涌现:迈向Web代理在真实世界中一致和透明的评估

Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

Emergence AI;Northwestern University(西北大学)

Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku

AI总结:

本文提出Emergence WebVoyager,通过明确的任务实例化、失败处理、注释和报告指南,提高Web代理评估的一致性和透明度,并展示了OpenAI Operator在不同领域和任务类型中的性能差异。

摘要:

可靠评估在复杂真实环境中运行的AI代理需要鲁棒、透明且与任务目标一致的方法。本研究指出现有AI代理评估实践的持续不足,特别是在Web代理评估中,如我们的WebVoyager审计所体现的,包括任务框架模糊和操作变量性,阻碍了有意义和可重复的性能比较。为解决这些挑战,我们引入Emergence WebVoyager,作为WebVoyager基准的增强版本,通过清晰的指南标准化评估方法。Emergence WebVoyager实现了95.9%的跨注释者一致率,表明任务制定和评估的清晰度和可靠性有所提高。将此框架应用于评估OpenAI Operator揭示了不同领域和任务类型中的显著性能差异,总体成功率为68.6%,明显低于OpenAI之前报告的87%,证明了我们方法在更严格和可比的Web代理评估中的实用性。

英文摘要:

Reliable evaluation of AI agents operating in complex, real-world environments requires methodologies that are robust, transparent, and contextually aligned with the tasks agents are intended to perform. This study identifies persistent shortcomings in existing AI agent evaluation practices that are particularly acute in web agent evaluation, as exemplified by our audit of WebVoyager, including task-framing ambiguity and operational variability that hinder meaningful and reproducible performance comparisons. To address these challenges, we introduce Emergence WebVoyager, an enhanced version of the WebVoyager benchmark that standardizes evaluation methodology through clear guidelines for task instantiation, failure handling, annotation, and reporting. Emergence WebVoyager achieves an inter-annotator agreement of 95.9\%, indicating improved clarity and reliability in both task formulation and evaluation. Applying this framework to evaluate OpenAI Operator reveals substantial performance variation across domains and task types, with an overall success rate of 68.6\%, substantially lower than the 87\% previously reported by OpenAI, demonstrating the utility of our approach for more rigorous and comparable web agent evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑