arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.02418cs.AIcs.LG
收藏

BrowserArena:在真实世界网页导航任务中评估LLM智能体

BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks

University of Pennsylvania(宾夕法尼亚大学)

Sagnik Anupam, Davis Brown, Shuo Li, Eric Wong, Hamed Hassani, Osbert Bastani

AI总结:

本文提出BrowserArena,一个基于真实开放网络的智能体评估平台,通过用户任务和逐步人类反馈识别出验证码、弹窗和直接导航三类失败模式,并揭示不同模型应对策略的差异与脆弱性。

摘要:

LLM网页智能体现在能够浏览开放网络并采取行动,然而当前的智能体评估局限于沙盒环境或人工任务。我们引入了BrowserArena,一个实时开放网络智能体评估平台,该平台收集用户提交的任务,进行竞技场式(Arena-style)的头对头比较,并使用逐步人类反馈来揭示失败模式。通过收集和分析智能体轨迹上的逐步注释,我们识别出三种一致的失败模式:验证码(captcha)解决、弹窗横幅移除以及直接导航到URL。通过构建针对性的数据集以进一步研究这些任务,我们发现了不同语言模型在应对这些失败模式时的差异。例如,我们发现o4-mini在规避验证码解决时采用了比其它模型更多样化的策略,而DeepSeek-R1在弹窗横幅关闭方面持续误导用户。我们的发现既揭示了当前网页智能体的多样性,也暴露了其脆弱性。更广泛地说,我们的基准测试方法提供了一种在大规模下评估和理解网页智能体失败模式的途径。

英文摘要:

LLM web agents now browse and take actions on the open web, yet current agent evaluations are constrained to sandboxed environments or artificial tasks. We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks, runs Arena-style head-to-head comparisons, and uses step-level human feedback to surface failure modes. Collecting and analyzing step-level annotations on the agent traces, we identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs. By constructing targeted datasets to further study these tasks, we discover variations in how different language models navigate these failure modes. We find, for example, that o4-mini deploys a wider variety of strategies to circumvent captcha resolution than other models and DeepSeek-R1 consistently misleads users about pop-up banner closure. Our findings surface both the diversity and brittleness of current web agents. More broadly, our benchmarking methodology provides an approach to evaluating and understanding web agent failure modes at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑