WebSuite:系统性地评估Web智能体为何失败
WebSuite: Systematically Evaluating Why Web Agents Fail
AI总结:
本文提出WebSuite,首个通用Web智能体诊断基准,通过Web动作分类法和可扩展任务套件系统评估智能体失败原因,将任务失败归因于特定动作失败,揭示改进方向。
摘要:
我们描述了WebSuite,这是首个面向通用Web智能体的诊断基准,旨在系统性地评估智能体为何失败。AI的进步催生了大量自主操作浏览器以完成任务的Web智能体。然而,大多数现有基准仅严格衡量智能体能否完成任务,而未深入揭示原因。在本文中,我们1)开发了Web动作分类法以促进识别常见失败模式,2)创建了一个可扩展的基准套件来评估智能体在我们分类的动作上的表现。该基准套件包含单个任务(如点击按钮)和端到端任务(如将商品加入购物车),其设计确保任务的任何失败都能直接归因于特定Web动作的失败。我们评估了两种流行的通用Web智能体(一种基于文本,一种多模态),并识别出每种智能体的独特弱点。由于WebSuite能将任务失败分解为特定动作失败,这实现了对单个智能体难以处理的UX流程的精细识别,并立即突出了改进的有希望途径。这些发现强调需要更聚焦地基准测试Web智能体出错之处,以有效提升智能体超越当前较弱的表现。
英文摘要:
We describe WebSuite, the first diagnostic benchmark for generalist web agents, designed to systematically evaluate why agents fail. Advances in AI have led to the rise of numerous web agents that autonomously operate a browser to complete tasks. However, most existing benchmarks focus on strictly measuring whether an agent can or cannot complete a task, without giving insight on why. In this paper, we 1) develop a taxonomy of web actions to facilitate identifying common failure patterns, and 2) create an extensible benchmark suite to assess agents' performance on our taxonomized actions. This benchmark suite consists of both individual tasks, such as clicking a button, and end-to-end tasks, such as adding an item to a cart, and is designed such that any failure of a task can be attributed directly to a failure of a specific web action. We evaluate two popular generalist web agents, one text-based and one multimodal, and identify unique weaknesses for each agent. Because WebSuite can disaggregate task failures into specific action failures, this enables granular identification of which UX flows an individual agent has trouble with and immediately highlights promising avenues for improvement. These findings highlight the need for more focused benchmarking on where web agents go wrong to effectively improve agents beyond their weaker performance today.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。